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Abstract. Combinatorial optimization problems such as finding submatrices with large 
average value within a large data matrix arise in a wide array of fields, ranging from 
statistical genetics, bioinformatics, computer science to various social sciences. These 
techniques play an important role in revealing substructures and associations with inter- 
esting characteristics in high dimensional problems. In this paper we analyze asymptotics 
for such problems in an idealized setting where the underlying matrix is a large Gauss- 
ian random matrix and provide detailed asymptotics for various characteristics of the 
energy landscape for such problems. For fixed k we provide a structure theorem for the 
k x k submatrix with the largest average. We then show that for any given r > 0, the 
size of the largest square sub-matrix with average bigger than r satisfies a two point 
concentration phenomena. Finding such submatrices for a fixed A; is a computationally 
intensive problem. We study the natural algorithm that attempts to find submatrices 
with large average; such algorithms typically converge to a local optimum. We prove a 
structure theorem for such locally optimal sub-matrices and derive refined asymptotics 
for the mean and the variance for L n (k) := number of such local optima. In particular 
for k = 2 and k = 3, the order of the means are n 2 and n 3 , while the variances are n 8 / 3 
and n 9//2 , respectively, with logarithmic corrections. We develop a new variant of Stein's 
method to prove a Gaussian Central Limit Theorem for L n (k) for all finite k. 



1. Introduction 

The study of random matrices, at the level of the empirical spectral distribution, has 
been one of the fundamental thrusts of modern probability theory. In the last few years, 
motivated by the explosion in data generated in biology especially genetics [24] , as well as 
complex networks [17], the problem of finding interesting structures or patterns within a 
large data matrix has become an important research direction. The simplest example of 
such structures are submatrices with large average. In the context of genetics, such subma- 
trices represent interesting patient to gene relationships, useful as a first step in identifying 
genes relevant to a disease (see [31] and the references therein). In the context of networks 
with matrices representing the strength of interaction between different individuals in the 
network, such submatrices (especially with the same row and column set) are thought to 
represent "communities" within the network. Finding submatrices with large average or 
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average above a certain threshold plays a crucial role in various exploratory analysis in 
such situations. 

Now suppose we are given a n x n square matrix W. Write [n] := {1, 2, . . . , n} for the 
row set (alternatively the column set) of W. Fix k ^ 2 and consider the task of finding 
the k x k submatrix amongst all possible (^) such submatrices with the largest average. 
For small k one can conceivably tabulate the average of all such submatrices, however 
the configuration space grows very quickly as k increases and such complete enumeration 
strategies start to become infeasible even for moderately large k in the context of the scale 
of data one has in practice. 

A number of iterative algorithms have been proposed to find such submatrices. One 
of the simplest such algorithms, often referred to as LAS (Large average submatrix, [31]) 
operates as follows. Represent a submatrix A as A := A x B where A, B C [n] represent the 
row and column set respectively of the submatrix. Start with a randomly chosen set Aq 
of k rows, find the k columns, say Bq with largest sums. Set Ao = Aq x Bq. Call this the 
column step; intuitively for a fixed set of rows we have found the "best" columns. Now 
proceed to the row step, where we keep the set of columns, namely -Bo, fixed and find the 
set of rows with the largest row sums say A\ and now let Ai = A\ x Bq. These two steps 
are iterated until one converges. This happens when the algorithm reaches a submatrix 
\* = A* x B* which is a local optimum, namely the minimal row sum of the submatrix 
A* is larger than the maximal row sum of ([n] \ A*) x B* and the minimal column sum of 
A* is larger than the maximal column sum of A* x ([n} \ B*). Empirical findings in [31] 
seemed to suggest that both in the context of empirical data as well as simulated data, 
this algorithm converged quickly and for real data, found matrices with interesting and 
interpretable structure. 

These empirical findings motivated us to provide a rigorous understanding of the general 
"energy" landscape of such problems in the simplest idealized setting where the underly- 
ing matrix W = ((wij))^^^^^ is a gaussian random matrix. Studying optimization 
problems and properties of optimal or locally optimal configurations for random data has 
now blossomed into a thriving subbranch of probability, see e.g. [32], [3] and particularly 
relevant to the kinds of questions addressed in this paper, see [15], [16] and [23]. In our 
context we are motivated by the following questions: 

(i) For a fixed k understand asymptotics for the average of the global optima as well 
as the structure of the optimal submatrix. Theorem 3.1 gives a description of these 
asymptotics. On the way we develop a new gaussian comparison result (Lemma 3.2) 
which is interesting in its own right. 

(ii) In the context of applications especially in the genome sciences, for fixed r, one 
is interested in finding the largest k := k(r) for which there is a k x k submatrix 
with average greater than this threshold r. We prove two point localization for k(r) 
(Theorem 3.4). 

(iii) For a fixed k understand asymptotics for the local optima of the LAS algorithm. 
The study of local optima in the context of exploratory data analysis has witnessed 
renewed interest over the last few years (see e.g. [25]). We give a complete structure 
theorem for the asymptotic distribution of a locally optimal matrix (Theorem 3.7). 
A simple corollary of this implies that asymptotically, the average of a typical local 
optimum is within a factor of 1/ \J2 of the global optima. 
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(iv) We study the number of local optima L n (k) (Theorem 3.8, 3.9). We derive refined 
bounds on the means and variances. These results heuristically suggest that the 
LAS algorithm converges in Bp^logn)^ -1 )/ 2 ) steps. More interestingly, the variance 
of L n (k) has non-standard scaling (for example for k = 2, E(L n (2)) ~ n 2 while 
Var(L„(2)) ~ n 8 / 3 ) . The reasons behind these results are the rather mysterious 
scaling of the positive correlations of pairs of locally optimal submatrices captured in 
Lemma 3.10 and Lemma 3.11. 

(v) We conclude by proving a central limit theorem for L n (k) (Theorem 3.12). In part 
due to the above non-standard scaling and highly complex dependency structure, 
current variants of Stein's method do not seem to apply to this situation and we 
develop a new variant of Stein's method suitable to this setting. 

1.1. Structure of the paper. The remaining paper is organized as follows. After a 
brief summary of notations used in this paper in Section 2, we present the main results 
in Section 3. We provide more background for the problems studied in this paper and 
connections between our work to existing literature in Section 4. We dive into proofs in 
Section 5 which collects some of the technical estimates we need for the proofs of the main 
results. The reader is urged to skim through these results and then come back to them 
as and when they are used. We complete the proofs about global optima in Section 6 and 
local optima in Section 7. Finally we present the proof of the central limit theorem for 
number of local optimal sub matrices in Section 8. 

2. Notation 

Given two integers a < b, define [a, b] := {a, a + 1, . . . , b — 1, b}. When a = 1, instead 
of [1,6] we will write [b] for simplicity. We will use bold alphabets, e.g. W, for denoting 
matrices and small alphabets, e.g. Wij, for denoting the corresponding entries. 

We shall construct all the finite n problems on the same probability space using a two- 
dimensional infinite array of i.i.d. standard Gaussian Random variables W := ((wij))i,j>i- 
For a fixed integer k > 1, let S^ n {k) be the collection of pairs of subsets of [n] := {1, 2, . . . , n} 
of size k, i.e., 

y n {k) := {I x J | I, J C [n], \I\ = \ J\ = k}. 

Note that \y n (k)\ = (£) 2 . For A, 7 G y n (k), we write |A n 7] = (s,t) if A and 7 share s 
many rows and t many columns. We will also write A n 7 = if | A n 7] = (s, t) with st = 0. 
In this case A, 7 are disjoint, they share no entries. 

For A = / x J £ J/' n (k), define as the sub-matrix ((u;jj))j g /j e j. For any matrix 
U = ((itjj)), we define 

avg(U) = lUr 1 ^^- 

as the average of the entries of the sub matrix U, here |U| is the number of entries of U. 
Define 

&(x) := P(Z < x) and <l>(x) := 1 - <fr(x) for x € E 

where Z is a standard Gaussian random variable. We shall use (") to denote the usual 
binomial coefficients and shall extend the definition for non-integer valued arguments using 
the Gamma function, in particular for any x £ [l,n], define 
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where T(a) := / °° x a 1 e x dx is the Gamma function. Note that for an integer k > 0, 
T(k + 1) = k\. Define the sequences 



a N := V21ogAf (2.2) 

and 

b N :=V2l^N- l ° S{4 / 7 l l0gN \ (2.3) 

V B 2^2 log N V ' 

These will arise respectively as scaling and centering constants in the statement of some 

of our results. Finally given any matrix U = ((ity)), we shall let m. denote the average of 

row i, u.j denote the average of column j and u.. = avg(W). We will write U = ((uij)) for 

the the Analysis of variance (ANOVA) decomposition of the matrix W = namely 

Uij = — Ui, — u.j + u„ (2-4) 
3. Main results 

We now state the main results in the paper. We start with asymptotics about the global 
optima in Section 3.1. We then analyze local optima in Section 3.2. 

3.1. Global optima. 

3.1.1. Structure of the Globally optimal sub-matrix. For an integer k > 1, define 

A* (A;) := argmax{avg(W A ) | A € y n (k)} 

M n {k) := max{avg(W A ) | A G ^ n (k)}. 

In particular, X*(k) is the row-column index for the globally optimal sub- matrix and M n (k) 
is the average of that matrix. We will prove a complete structure theorem for W^*^) in 

Theorem 3.1. Recall that for a fixed k there are N = (^) 2 many square submatrices of 
size k. The first part of Theorem 3.1 says that the global optimal average has the same 
distributional asymptotics as that of the maximum of N i.i.d. Gaussian random variables 
as long as k := k{n) ^ c log nj log log n. We believe the result should be true as long 
as k logn, however this extension will require new ideas. The second part of the 
theorem says that to first order, such first order asymptotics continue to hold as long as 
log/c <ti logn. The third part gives a structure theorem for the actual matrix X*(k) for 
fixed k. 

Theorem 3.1. Let N = (^) 2 and recall the constants ajy and bj^ from (2.2) and (2.3). 

(a) There exists a constant c > such that for k < c logn/ log logn we have 

a N {kM n {k)-b N ) =U -logT 

as n — > oo where T ~ Exp(l). 

(b) In general, for c logn/ log logn <k< exp(o(logn)) we have 

P(-fcw„(log logn) 2 / logn) < a N (kM n (k) - b N ) < uj n ) 1 

for any function oj n — > oo as n — )• oo . 

(c) Moreover, for each fixed integer k ^ 1 we have 

W A . (fc) - avg(W A * (fc) )ll / W[ fc ] x [ fc ] - avg(W Wx[fc ])ll' 
where 1 is the column vector of all ones. 
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Our main ingredient will be the following comparison Lemma, which is of independent 
interest. 



Lemma 3.2. Fix N 2 and let (Xi, X2, ■ ■ ■ , -Xjv) be jointly Gaussian with E(Xj) = 
0,Epf?) = 1 and E(X i X j ) = <j y e (-1,1) for 1 < i < j < N. Let Z U Z 2 ,...,Z N be 
i.i.d. standard Gaussian random varaibles. For any u > 1, we have 

I P( max Xi < u) — P( max Zj < it) I 

l<i<iV l<i<N 

< 2min{l, |1 — 9ij\u(u + l)}$(it)$((l A %)u) 



where jV(u) = X^j=i ^ u l an d the first inequality is by Slepian's lemma. Various 

gaussian comparison results similar to Lemma 3.2 are known (see [7,18,22]). This vari- 
ant seems best suited for our purposes and in particular allows us to extend first order 
asymptotics for k = k(n) — > 00 as in Theorem 3.1 (b). 

3.1.2. Two-point localization. Fix r > 0. Let M* = M*{t) denote the largest k such that 
there exists a k x k sub matrix Wa with A G y n {k) and avg(W>,) > r. The next theorem 
states that for each fixed r > 0, the random variable M*(t) localizes on at most two 
consecutive values as n — > 00. 

Recall the definition of the binomial coefficient for non-integer values from (2.1). Let 
k = k n {r) > 1 denote the unique solution (for large n) of the equation 



l<i<j<N 




where 9ij = ^/(l — o~ij) /(l + cry) and x + = max{x,0}. 
Note that as a corollary, when o~ij > for all i, j, we have 



< P( max Xi<u)- P( max Zi < u) < E(AA(n) 2 ) - N$(u) 



Ki<N ' Ki<N 



2 




(3.1) 



It is easy to check (see e.g. [34]) that 




(3.2) 



as n — > 00. Let k* denote the closest integer to A:. In [34] it was proved that 
Theorem 3.3 (Theorem 1 in [34]). For fixed r > 0, we have 




as n — >■ 00 . 



We improve the result to a two point localization result. 
Theorem 3.4 (Localization for fixed threshold r). We have 

F(M*{r) = k* or k* -1)^-1 



as n — >■ 00 . 
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3.2. Locally optimal sub matrices. Fix k > 1 and recall the algorithm described in 
Section 1 designed to detect kxk sub-matrices with large average. Note that this algorithm 
terminates at a local optima. By definition a submatrix is a local optima if it is an optima 
in the column step and row step of the algorithm. The next few results give asymptotics 
for the distribution of a typical local optima as well as the number of such local optima. 
For future reference we first formalize these definitions. 

Definition 3.5. Given two subsets A = / x J € y n {k), we call the sub-matrix := 
(( w ij))iel,jeJ row optimal if 

avg(W /xJ ) = max avg(W /xJ /) 

\J'\=k 

and column optimal if 

avg(W /xJ ) = max avg(W 7 / xJ ) 
\i'\=k 

A submatrix which is both row and column optimal is called locally optimal. 

We are interested in the distribution of conditioned on being a local optima (by 
symmetry the choice of A is irrelevant) as well as the number of such local optima namely, 

L n (k) '■= 1{W^ is locally optimal}, (3-3) 

\e.y n (k) 

the number of k X k locally optimal submatrices. Note that, for any fixed set of k rows 
I C. [n], there is a unique row optimal sub matrix W/ X j*(/). Thus we can also write 

L n (k) = 5^ 1{Wj x j«(j) is column optimal }. (3-4) 
\i\=k 

From (3.4) it is easy to see that 

E(L n (£;)) = (j^j ^(^A fe is column optimal | is row optimal) 

where A& = [k] x [k] £ y n {k). Thus our first order of business is understanding the 
asymptotic conditional probability of a submatrix being a column optimal given it is row 
optimal; by symmetry this is the same for all k x k submatrices. Intuitively one might 
expect P(W^ fc is column optimal | W^ fc is row optimal) = 0(1) as n — > oo since in some 
sense we have already conditioned the entries of W^ fc to be large. Indeed for k = 1, we 
have P(W^ 1 is column optimal | is row optimal ) = n/{2n — 1) ~ 1/2. However, it 

turns out that, for k > 2, the conditional probability that a row-optimal matrix is also 
column optimal vanishes as n — > oo and in fact behaves like (log n)~( fe_1 )/ 2 . We first give 
some intuition behind this phenomenon and then state a precise structure theorem for the 
distribution of a local optima. We will need the following standard result from extreme 
value theory, see e.g. [21]. 

Lemma 3.6. Let Z±, Z2, ■ ■ ■ , Z n ben i.i.d. standard Gaussian random variables and Zn\ < 
Z{2) < ■ • < -^(n) be their ordered values. Recall the scaling and centering constants a n ,b n 
from (2.2) and (2.3). Then for any fixed integer £ > 0, we have 

0"n(Z(n) - °n, Z(n-1) ~ K, ■ ■ ■ > Z {n-t) ~ K) => (Vl, V 2 , . . . , V() 
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as n 



oo where Vi = — log(Ti + Ti + • • • + Tj), % > 1 and Ti 's are i.i.d. Exponential^ 



random variables. Moreover, the point process 
Poisson Point Process on R with intensity e~ x . 



,i Oa n {Z,-b n ) converges as n 



oo to a 



Using Lemma 3.6 one can see that all the column averages of a row optimal matrix 
U (and hence the matrix average avg(U)) will be concentrated around k~ x l 2 b n with 
0(1 /a n ) fluctuation. Recall that a n = y/2 logn. Now for a Gaussian random matrix 
with i.i.d. entries the centered row averages are independent of the matrix average. Hence 
the minimum row average will be the same as the matrix average avg(U) — V, where 

V = k~ 1 / 2 max 1 < i <fc(Z — Zj). Here {Zi} 1<i<k are i.i.d. standard Gaussian r.v.s. Note that 
we have fixed = [k] x [k]. Fixing this column set, the maximum of all other row averages 
(amongst rows [ti] \ [&]), by Lemma 3.6, will be concentrated around k ^^ 2 b n with 0(1 j '&ri) 
fluctuations. In order for the row optimal matrix to now be column optimal, one needs V 
to be of the order of l/a n , the probability of which turns out to be of order 1/a^ -1 as the 
vector [Z — Z\ : Z — Z%, ... ,Z — Z^) lies in a (k — l)-dimensional subspace. 

Let Ik be the event that WftwiH is locally optimal. We will prove the following structure 
theorem. 

Theorem 3.7 (Structure Theorem for locally optimal submatrices). Conditional on the 
event Ik, the sub matrix Wny x [fc] can be written as 



W 



[*]x[k] 



\fk Vka n 

log(l + T/G) 
Vka n 



11' + Z 

kUi - 
kU 2 - 



kU k -l 



r | iog(i + r7G) i 

Vka n 



kU> 2 



kU' k 



+ o p (l/a n ) 



where Z = (zij)kxk with z%j = Zij — Zi. — z.j + z.., Zij 's are i.i.d. standard Gaussian, U = 
(Ui, U2, ■ ■ ■ , Uk), U' = (U[,U2, ■ ■ ■ ,U' k ) are i.i.d. from Dirichlet(l, 1, . . . , 1) distribution 
independent of (G, T, T') which have joint density 



k-l fc-l -t-t'-2g 



<x(log(l + t/g)log(l + t'/g)) k - 1 g k - 1 e 
Further, there exists a real number 81. > such that we have 



g,t,t'>0. 



P(Xfc) 



0(logn)( fc -D/ 2 
The value of Ok can be explicitly computed as 

£.2fc+l/2 



(1 + o(l)) as n — y 00. 



2 2fc-l 7r (fc-l)/2jfe|2 



E((iog(i + y/G)io g (i + y7G)) 



(3.5) 



where Y,Y' are i.i.d. Exp(l) and G ~ Gamma(fc, 2) with density ^ k 2 _ 1 y X k x e 2x ,x > 
independent of Y,Y' (see equation (7.6)). 

Note that as a corollary we have, all the entries in a typical locally optimal sub matrix 
are concentrated at y/2 \ogn/k(l + o(l). By symmetry, the expectation of the number of 

local maxima L n (k) is E(L n (fc)) = (^) 2 P(Xfc). Thus theorem 3.7 immediately yields the 
following result. 
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Theorem 3.8 (Mean behavior). For any fixed k >1, the expected number of local maxima 
scales like 

HLn{k)) = y_ 1)/2 (l + o(l)) as n -> oo. 

Here 9k > is as in (3.5). 

Intuitively this suggests that the running time of the LAS algorithm can be bounded 
by a Geometric random variable with p = p(n) = 9 k /(logn)^ k ~ 1 ^ 2 , and thus converges in 
0p((logn)( fc-1 )/ 2 ) steps, and thus gives conceptual insight on empirical observations on 
the running time of the algorithm. 

The behavior of the variance of L n (k) is much more delicate and requires analyzing 
the joint distribution of two locally optimal sub-matrices and in particular yields non- 
standard scaling as described in the Introduction, in particular \ax(L n (k)) >> E(L ra (/c)). 
The reason behind the high value for the variance is the complex dependence structure 
amongst locally optimal matrices. 



Theorem 3.9 (Variance behavior). There exists v\. 6 (0,oo) such that 

2 



,2fe 2 /(fe+l) 



Var(L n (fc)) = (lQgre)fc2/(fc+1) (l + o(l)) as n 



oo. 



There are two main ingredients in this variance calculation which are of independent 
interest and could conceivably be of use in the analysis of similar iterative methods. The 
first is the following lemma which gives the probability that both the maximum row average 
and maximum column average of a Gaussian random matrix with i.i.d. entries are large. 

Lemma 3.10. Let U be a s x t matrix of i.i.d. standard Gaussian entries. Recall that 
Ui.,u.j was respectively the i-th row average and j-th column average o/U. For any fixed 
9 > 0, x, y G R, we have 

P( max m. > 9b n + x/a n , max u.j > 9b n + y/a n ) 
l<i<s i<j<t 



4-o/u i\ i I i\ \ll i 1\ atis-l-t— i)t>- st{s+t-2)e , 

= (rj(s,t,e) +o{l))e- stm - 1)x+{s - l)y)/ ( st - 1) n ^^^(logra) 2 ( st ~ 1 ' 1 

for some constant ij(s,t,9) > 0. 

The heuristic idea behind the proof of Lemma 3.10 is the following. If both the maximum 
row average and maximum column average are bigger than z, there will be at least one 
row (say i*-th row) and one column (say j*-th column) with average bigger that z. The 
joint density of the i*-th row and j*-th column is proportional to exp(— (Yli u ij* + 
X^vy* u l t j + M ljJ/ 2 )- !f we minimize Y^i^i, u %* + ^j¥=j* u i*j + u i*j* un der the constraint 
that Yi u ij* ^ Ylj u i*j — sz i the minimum is achieved at 

(st-s)z . . (st-t)z 

u ij* = — : — T~ for %'» = — : — T~ toT 3 T 3* 

st — 1 st — 1 

(2st - s-t)z 
and Ui.<i* = • 

LJ. st _ 1 

Plugging in these values in the exponent results in the value st(s + t — 2)z 2 / (st — 1). When 
z = 9b n , we have 

st(s + t-2)z 2 



■i 



6XP( 2(st-l) ] " n 
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which is the leading order in the probability. The complete proof is given in Section 5. 

The other ingredient is the following joint probability estimate for two locally optimal 
matrices. We will need to setup some notation. Fix two integers s,t £ [k]. Let B s j,k be 
the event that ^W[k]x[k] is locally optimal as a sub matrix of W[k]u[s+k+i,n]x[k]u[t+k+l,n] and 
w [ s +i, s +fe]x[i+i,t+fe] is locally optimal as a sub matrix of W[ s+ljn]x[t+lin ] (see Figure 3.1). 



Xi 


x 2 




x 3 


x 4 


x 5 




x 6 


x 7 



[1,6 



[k + l,k + s]< 



[k + s+1, n]l 



[l,f] [t+l,k}[k + l,k + t] 



[k + t + l,n] 



W[ tM4 



W[»+l,«+*lx[(+l,i+<:] 



W [t . +i+1 



w. 



[1 H 



,n]x[(+lJ+t 



w 



[k]x[fc+t+l,n] 



W[ 4+ l, s+t ]x[t+(+l,, 



Figure 3.1. A pictorial representation of the event B s j,k and the block 
matrices Xj, 1 < i < 7. 

Lemma 3.11. Let < s < k,0 < t < k. There exists a constant r](s, t,k) > such that 

2k-2k(k-s)(k-t)/(2k 2 -st) 

P(B Stt , k )<r](s,t,k) 



\/logn 



It clearly follows that the joint probability T(B Si t,k) is much bigger than the product 
of the probabilities that both W^,] x [^] and ^W[ s +i, s +k]x[t+l,t+k] are locally optimal. Now 
note the block matrix decomposition shown in Figure 3.1. Under the event B St t t k, the 
average entry in X4,Xi,X7 are much larger than the average entries in X2,X3,Xs,X6. 
However the global average is still close to y / 21ogn/A; which is the average of a typical 
locally optimal matrix. 

Finally, using Stein's method we will prove asymptotic normality for the random variable 
L n {k). Despite being able to express L n as a sum of random variables 

L n (k) ■= y 1 {W A is locally optimal in W[ n ] x[n] } , 
\ey n (k) 

as opposed to typical settings of weak dependence, here the event that a particular sub- 
matrix Wa is locally optimal affects every other submatrix A' G y n {k). The analysis of 
Var(L n (/c)) suggests non-trivial correlations. This makes the analysis of the asymptotic 
distribution particularly involved. 

Recall that the Wasserstein distance between two random variables W, Z is defined as 

d W (W,Z) :=sup{|E((7(^))-E( 5 (Z))| 1-Lipschitz} 

The following result quantifies the distance from normality of L n (k). 
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Theorem 3.12 (Central Limit Theorem for L n (k)). We have, 

f L„ W -E(L„ W ) d , 

VVar(L„(t)) 

as n —)• oo. Moreover, we have 

d w (L n (k),N(0,l)) < n -(^^ +0(logl ° sn/logn) . 

where dyy is the Wasserstein distance between two distributions. 

Here we mention that the rate of convergence in Theorem 3.12 is definitely not optimal 
and we haven't tried to find the optimal rate. However, we include simulation results in 
Figure 3.2 for k = 2 and n £ {100,200} with 5000 runs to show the fast convergence to 
Gaussian limit empirically. 




450 500 550 GOO -4 -2 2 4 

Number of local optima Theoretical Quaotiles 




1800 1900 2000 2100 2200 -4 -2 2 4 

Number of local optima Theoretical Quantiles 



Figure 3.2. Histogram and QQPlot for number of local optima for k = 2 
with n = 100 (top row) and n = 200 (bottom row) with 5000 samples. 



4. Discussion 

We now discuss the relevance of these results and related work. We start with a dis- 
cussion of the general detection problem considered in this work and then expand on the 
techniques used in the paper. 
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4.1. Finding large substructures. As mentioned above, with the advent of large scale 
data in genomics, problems such as finding interesting structures in matrices has stimulated 
a lot of interest in a number of different communities, see e.g. the survey [24]. In spirit, 
such problems are linked to another large body of work in the combinatorics community, 
namely the hidden clique problem sec e.g. [28] or [19] and the references therein. The 
simplest statement of the problem is as follows: Select a graph at random on n vertices; 
consider the problem of detecting the largest clique (fully connected subgraph) . For large 
n, it is known that the largest clique has k(n) ~ 21og 2 n vertices ([8], [9]). Theorem 3.4 
is very similar, in spirit to this result. However most greedy heuristics and formulated 
algorithms, short of complete enumeration, are only able to find cliques of size ~ log 2 n 
and thus are off by a factor of 2 from the optimal size. We see analogous behavior in our 
results; Theorem 3.1(a) implies that for fixed k, the average of the global optimum scales 
like \/2y/\ogn/k whilst Theorem 3.7 implies that the average of a typical local optima 
scales like y/logn/k. 

4.2. Planted detection problems. In the context of statistical testing of hypothesis, 
we have analyzed the energy landscape in the "null" case. One could also look at the 
"alternative" where there is some inherent structure in the data. In the last few years 
there has been a lot of interest in formulating statistical tests of hypothesis to distinguish 
between the null and the alternative, see e.g. [5] and [6] for the general framework as well as 
application areas motivating such questions and see [1] and [10] for a number of interesting 
general results in these contexts. In the context of the combinatorics, such questions result 
in the famous planted clique problem see e.g [4], [13] and the references therein. 

4.3. Energy landscapes. The notion of energy or fitness landscapes, incorporating a 
fitness or score to each element in a configuration and then exploring the ruggedness of 
the subsequent landscape, arose in evolutionary biology, see [37], and for a nice survey, 
see [29]. Our work has been partially inspired by the rigorous analysis of the NK fitness 
model ([20], [35]) carried out in the probability community in papers such as [15], [16], [23]. 
These questions have also played a major role in understanding deep underlying structures 
in spin glass in statistical physics, see e.g. [27]. For general modern accounts of the state 
of the art on combinatorial optimization in the context of random data and connections 
to other phenomenon in statistical physics, we refer the interested reader to [26]. 

4.4. Stein's method for normal approximations. Stein's method is a general and 
powerful method for proving distributional convergence with explicit rate of convergence. 
Developed by Charles Stein in 1972 [33], to prove Gaussian central limit theorems for sums 
of random variables with complex dependency structure. This has now been extended to 
a wide array of distributions. Here we briefly discuss the case of normal approximation. 

The standard Gaussian distribution can be characterized by the operator Af(x) := 
xf(x) — f'(x) in the sense that, X has standard Gaussian distribution iff W,(Af(X)) = for 
all absolutely continuous functions /. Now to measure the closeness between a distribution 
v and the standard Gaussian distribution vq, one generally uses a separating class of 
functions T> to define a distance 

d v (v,v ) = sup|E/i(X) -Eh(Z)\ 
hev 
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where X ~ z/, Z ~ N(0, 1) and then attempts to show that the distance is "small". In 
this paper we will consider the L 1 -Wasserstein distance in which case T> is the class of all 
1-Lipschitz functions. 

Stein's method consists of two main steps. The first step is to find solution to the 
equation Afh(x) = h(x) — ~Eh(Z) for h E D. Assuming this can be performed, we have, 

sup|E/i(X) -E/i(Z)| < sup |E(X/(X) - f'(X))\ 

heV /££>' 

where T>' = {fh \ h £ T>}. The following lemma summarizes the the bounds required for 
Stein's method. 

Lemma 4.1 ([33]). For any 1-Lipschitz function h, there is a unique function f^ such that 
■Afh = h — E h(Z). Moreover we have 

\fh\oo < 1, \fh\oo < VV^ and \ fh\oo < 2. 

Thus to prove that the distribution of X is close to standard Gaussian distribution it is 
enough to prove that 

sup \ Ef'(X)-EXf(X)\ 

few 

is small where 

V = {/ I l/fcloo < 1, \f' h \oo < VV^ and l/^'loo < 2}. (4.1) 

This final portion is very much problem dependent and is often the hardest to accom- 
plish. A number of general techniques have now been formulated, e.g., exchangeable pair 
approach, dependency graph approach, size-bias transform, zero-bias transform etc. that 
can be used for a large class of problems. We refer the interested reader to the surveys 
[11, 12, 14,30] and the references therein. However, in our case because of the high degree 
of dependency, the above mentioned methods are difficult to apply and we develop a new 
variant to bound the error. 

4.5. Open questions. For the sake of mathematical tractability, we assumed that the 
underlying matrix had gaussian entries. It would be interesting to extend this analysis to 
general distributions. The exact statement of the results will be different since extremal 
properties of the gaussian distribution play a significant role in the proofs of the main 
results. The results in the paper also suggest a host of extensions and new problems. 
Theorem 3.1 deals with the global optimum in the regime where log k = o(logn). Extending 
this further, especially to the regime where k = an for some < a < 1 would be quite 
interesting and will require new ideas; one expects that the comparison to the independence 
regime using Lemma 3.2 breaks down at this stage. We also expect behavior similar to the 
extrema of branching random walk ([2] and references within) in this regime. Extending 
the local optima results to a regime k = k(n) — > oo as opposed to the fixed k regime would 
be interesting. This would be especially relevant in the context of detecting matrices with 
average above a particular threshold which by Theorem 3.4 corresponds to the k(n) = 
Clogn regime. Finally this work fixes k and then tries to find submatrices with large 
average. It would be interesting to develop algorithms which allow one to increase k to 
achieve large submatrices with average above a threshold r. 
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5. Technical Estimates 

We start with some technical estimates that will be needed in the later proofs. We start 
with various estimates on the tails of the normal distribution which will then lead to a 
proof of Lemma 3.10 in Section 5.3. We conclude this section with some combinatorial 
estimates. 



5.1. Gaussian tail bounds. The following is a standard bound on the Gaussian tail, see 
e.g. [21]. 

Lemma 5.1. Let Z be a standard Gaussian r.v. Then we have 

< P(Z > x) < for x > 0. 



V2^(l + x 2 ) V^ttx 

The next result uses the above Lemma to understand the conditional distribution of a 
standard Gaussian conditioned to be large. 

Lemma 5.2. Let Z be a standard Gaussian random variable and 6 > be a fixed real 
number. Let a n and b n be as in (2.2) and (2.3). Let B x be the event {Z > V0(b n + a^x)} . 

(a) We have 



n 



e (\/2Trb n ) 1 e e x8 F(B x ) converges to 8 l l 2 uniformly for \x\ < a % 



as n — > oo . 

(b) Conditionally on the event B x , a n {Z/\/6 — b n — a~ l x) converges in distribution to an 
Exponential rate 9 random variable as n —> oo for all ieft. 



Proof, (a) We will use Lemma 5.1 and the fact that n(v / 2vr& n ) 1 e b ™/ 2 — > 1 as n — > oo. It 
clearly follows that 



lim n e {V2^b n ) l - e e £d V{B x ) 

n— >oo 

= lim ra°(v&n) 1_ V fl P(Z > Vdibn + a^x)) 

n— >oo 

= lim J_ n e {V2Kb n ) l ~ e e xd exp(-8(b n + a^ 1 x) 2 /2) = 9~ l/2 . 
(b) Now for the conditional distribution note that 

P(a n (Z/V9 -b n - a^x) > t | B x ) = ^kj -> e" w 

as n — > oo for every t > 0. This completes the proof. ■ 

To analyze asymptotics for the expected number of local optima E(L n (/c)), we will need 
to understand the distribution of the deviations of a set of Gaussian random variables 
from the sample mean under various conditioning events. The next Lemma quantifies the 
results relevant to our treatment. 

Lemma 5.3. Let Zi, Z%, . . . , Zk be i.i.d. standard Gaussian r.v.s. Let Z = ^ ls [^l = \Zi 
and Z m [ n = minK^jt Zi be, respectively, the average and minimum of the random vector 
(Zi, Z2, ■ ■ ■ , Zk). 
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(a) The random variable Z — Z mm > a.s. and we have P(Z — Z mm < x) = f k x k ~ 1 (l + 
o(l)) as x 10 where 

fk := A;!(2vr)( fc - 1 )/ 2 ' 

(b) Let B £ be the event that {Z — Z m [ n < e} for e > 0. The conditional distribution 
of e~ l {Z — Z\, Z — Z2, ■ ■ ■ , Z — Z k ) given B £ converges in distribution, as e | 0, to 
(1 — kUi, 1 — kU2, . . . , 1 — kU k ) where U = (JJ\, U 2 , ■ ■ ■ , U k ) follows the Dirichlet dis- 
tribution with parameter (1,1,..., 1), i.e., U is uniformly distributed on the simplex 
{{xi, x 2 , ■ ■ . , x k ) I x\ + x 2 H h x k = 1, x\ > 0, . . . , x k > 0}. 

(c) We have 

P(Z - Z min >x) = ^r exp (- 2^^ ) (! + o(l)) «» t 00 

for some real number g k > 0. 

Proof of Lemma 5.3. (a) The first assertion that Z — Z m [ Q > follows trivially. It is easy 
to see that /c _1 (Z max - Z min ) < Z - Z min < Z max - Z min where Z max = maxi<j< fe Zj. In 
particular, 

P(Z max - Z min <x)< P(Z - Z min <x)< P(Z max - Z min < kx) for all x > 0. 
Now one can easily check that 

/oo 
k(^(t + x)-^(t)) k - 1 ^>\t)dt 
-00 

where $(t) = ~P(Z < t) is the standard normal distribution function and the right hand 
side is 0(x fc_1 ) as x | 0. This proves that F(x) := P(Z — Z mm < x) scales like x k ~ x as 
x i 0. Now Z — Z m i n has a density on P + and its Laplace transform is given by 

E(e^- Z ™)) = ke~ t2/2k [ e- tx ^(x) k - 1 ^'(x)dx 

= ke {k - 1)t2/2k [ $(x - t) k - 1 ^>'(x)dx for t > 0. (5.1) 

The proof follows from the fact that E(e*^) = e* 2 / 2fc , E(e* Zmin ) = f R ke~ tx $>(x) k ~ 1 $'(x)dx 
and by independence of Z, Z — Z m ; n we have E(e~*( z ~ Zmin )) = E(e iZmin )/E(e* z ). From 
our previous calculations it thus follows that there is a constant f k > such that 
F(x)/ f k x k ~ l -> 1 as 1 1 0. We claim that 

f k = lim 77 t ^T,E(e- t ^- z - i «)). 
t->oo (fc — lj! 

We leave the proof to the interested reader. Using (5.1) and the fact that 
$(-x) = —^e- x2/2 (l - 0(x- 2 )) for x -»• 00 
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we finally have 

' k ~t%£> A:!(2vr) fc /2 J R \ t-x 

k 2 f f t(l + 0((t-x)- 2 )) 
t^oo k\{2n) k l 2 J R \t/k - (x - (1 - l/fc)i) 

= A;!(2vr)( fc - 1 )/ 2 ' 

(b) For fixed e > write P £ for the conditional distribution of e~ l {Z — Z\,Z — 
Zi y ...,Z — Zk) given B £ := {Z — Z m { n < e}. For any e > 0, the support of P £ is 
the set Afc := {(x±,X2, ■ ■ ■ , Xfc) | xi + X2 + • • • + Xk = 0, x\ < 1, . . . , Xk < 1} which is 
a simplex with corner points v% := (1, . . . , 1, 1 — k, 1, . . . , 1) where 1 — k is in the i-th 
position for i = 1,2, ... ,k. The distribution is coordinate permutation invariant for 
each e > 0. Obviously {P e : e > 0} is a tight family of probability measures on M fc . 
Every subsequential limit of P E as £ | is translation invariant within the simplex. 
Hence the limiting distribution exists and is uniform over the simplex Now given a 
Dirichlet(l, 1, . . . , 1) random vector U = (U\, U2, ■ ■ ■ , U^) one can get a uniform random 
point from the simplex by taking ^i v % = (1 — kUi, 1 — kU2, . . . , 1 — kUk). Thus 
we are done. 

(c) From the fact that Z — Z m [ n = maxi<j<fc{Z — Zi} it is easy to see that 

P(Z -Zx>x)< P{Z - Z min >x)< k¥(Z -Zx> x). 

The rest follows from the fact Z — Z\ is normal with mean zero and variance (k — l)/k, 
and for a standard Gaussian random variable Z, 

-x 2 /2 

l?(Z>x)= = (l + o(l)) as x too. ■ 

5.2. Maxima of two correlated gaussian r.v.s. Let (Z, Z„) be a bivariate gaussian 
random vector with E(Z) = E(Z p ) = 0, Var(Z) = Var(Z p ) = 1 and E(ZZ p ) = p > 0. 
Note that, we can explicitly construct such a distribution by taking Z be standard normal 
and constructing Z p = pZ + y 1 — p 2 Z' where Z' is an i.i.d. copy of Z. To estimate the 
conditional probability P(Z p > x \ Z > x) for 'large' x, first of all note that conditional 
on A = {Z > x}, the random variable x(Z — x) remains tight (in fact, using an argument 
similar to the proof of Lemma 5.2(b), one can show it converges to Exp(l) distribution as 
x — > 00) and thus Z is concentrated around x conditional on A. Now conditionally on A, 
the event {pZ+\Jl — p 2 Z' > x} is roughly equivalent to the event { \J\ — p 2 Z' > (1— p)x}, 
which has probability &(0x) where 9 = — p)/(l + p). The following Lemma 5.4 makes 
these ideas precise. 

Lemma 5.4 (eqn. (1.2) in [36]). Let Z,Z' be two i.i.d. standard Gaussian r.v.s. Then, for 
any p E [0, 1] and x > we have 

§(0x) < P(pZ + v 7 ! - P 2 Z' > x\Z > x)^{l + p)${6x) 
where 




r \((k-l)(x~tf+x^) d2 



r l ix - (1 -x /k) t)* dx 
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5.3. Proof of Lemma 3.10. Recall that Lemma 3.10 computed the asymptotic proba- 
bility of the maximum row and column sum of a rectangular matrix being large simulta- 
neously. Using the tail bounds in the previous Section, let us now prove this Lemma. 

Proof. Let Z, Z±, Z%, . . . , Z s , Z[, Z' 2 , . . . , Z[ be i.i.d. standard Gaussian r.v.s. Define 

V s = max (Zi - Z), V[ = max (Z- - Z'). 

l<i<s 1<J<* 

It is easy to see that 

( max Ui. — u.., max u.j — u..,u..) = (t~ l l 2 V s , s~ 1 ^ 2 V^ , (st)~ lj/2 Z). 

l<i<s l<j<t 



Define a n = y/si(9b n + x/a n ) and f3 n = \/si(9b n + y/a n ). Hence we have 
p := P( max U{. > 9b n + x/a n , max u.j > 9b n + y/a n ) 

l<i<s 1<7<* 
= P(K > (a n - Z)/y/i, Vl > (Pn - Z)/Vt). 

Now using Lemma 5.3(c) we have 

p = (Vst 9s g t + o(l)) e( {a n - Z)-\p n - Z)^ exp ( K " Z? {Pn ~ Z? 



2(s-l) 2(t - 1) 

where the expectation is w.r.t. Z. One can easily check that 

(a n -z) 2 {Pn-zf 2 
s-1 t-l 

st-l ( z (t-l)a n + (s-l)/3 n \ 2 | ta 2 n + sl3 2 n -2a n l3 n 



{s - l)(t - 1) V st ~ 1 / st ~ 1 

Define 

q '= 6XP { 2(^-1) 

Recalling that a n ,b n ~ y/1 logn, we have 

q (st-i) 2 (Vs-t 9s9t + o(i)) f f (St-l)z 2 

(s - l)(t - l)(ta n - Pn)( S p n - a n ) J R y \ 2(a-l)(t-l)' 
^ (rj(s,t,9) + o(l)) 
log n 

for some constant rj(s,t,9) > 0. Finally note that, 

q = exp ^«t-l)x+(8-l)y)j ^^n } (1 + 

and this completes the proof. 
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5.4. Combinatorial estimates. We start with the following Lemma on binomial coeffi- 
cients which easily follows from Stirling's approximation. 

Lemma 5.5. For any n, 1 < k < \fn we have 



(n)k 



e -k 2 /2n+0(k/n) 



The next shows the asymptotic negligibility of a particular series which arises in deriving 
results about the global optima. 

Lemma 5.6. (a) Let N = (™) 2 , a N = y/TtogN , b N = v / 21ogiV- log(4vr log N)/2y/T\ogN 
and un = bjy — x/ajy. There exists a universal constant c > 0, such that for k < 
c log nj log log n and any fixed x £ H, we have 



V l k \( k \( n - k \( n - k \( n \ 2 j k2 + St c stul/(k* + st) )Q 



' \s J \ t J \ k — s J \ k — t J \ k J V k 2 — st 

l<s,t<k 
st^k 2 

as n — ^ oo. 

(b) The same result holds iflogk <C logn and A;(loglogn) 2 /logn <C x <C a N . 

Proof of Lemma 5.6. Throughout the analysis we will assume that k = e o( - l °£ n \ Let N = 
(I) 2 and define a N = ^2\og N,b N = ^2 log N - log(4vr log N) /2^2 log N. 

Let un = bN — x n /aN with —K<x n <^a N for some constant K > 1. Clearly we 
have e u Jv/ 2 = Ne~ Xnrn+0 ^' /y/2iraN where r n = 1 + x n /2a? N — > 1 as n — > oo. Moreover 
using Stirling's approximation and the fact that k <C \fn one can easily see that N = 
^^jp- (en / k) 2k . Thus we have 

e*/*<^(«£=) (5.2) 



A: 



\J k 3 log 



n 



for some universal constant Co > 0. Now note that (k 2 + st)/(k 2 — st) < k for all integers 
s, t with 2<s + t<2k — 1. Thus we need to show that 



as n — )• oo. 
Note that 



v s/ W/ \k — s J \k — t J \k 

l<s,t<k,st^k 2 /v /v 



n- k\ fn\ 1 {k) s (n) 2 k- 



k — s ) \ k ) (n)| 



where (n)& := nl/(n — k)\. Thus fc <C y^n and Lemma 5.5 imply that 

n — ti\ fn — fc\ f n\ 2 c(k) s (k)t 



K k — s J \k — t J \k J n s+t 
for some universal constant c > 0. Using Stirling's formula once more, we have 

(k) s (k) t < k(k/e) s+t e~ k{1 ~ s/t)log(1 ~ s/k) ~ k(1 ~ t/k)l ° e(1 ~ t/k) \ 
Moreover the function f(x) = —(1 — x) log(l — x) is concave. Thus we have 

(k) s (k)t < k(k/e) s+t e 2kf ^ s+t ^ 2k \ 
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Note that Ast < (s + t) 2 implies that 

st (s + t) 2 

k 2 + st ~ Ak 2 + (s + t) 2 ' 



Thus 



l<s,t<k 
st^k 2 



u 2 N {s + t) 2 
Ak 2 + (s + 1) 2 J ' 



Now 

l 



s=0 

for all integer I where h{x) = — xlogx — (1 — x) log(l — x). Thus we have 

2k -^^ 1 / n\ u 2 N i 2 



In < ck 3 / 2 ex P ( 2fc 5 



2k J Ak 2 + l 2 , 

where g(x) = —xlogx — 2(1 — x) log(l — x). Using Equation (5.2) we have 



< 



c e XnTn 



l/2k 



en y^/k 3 logn / 

and thus finally we have 



l n < Cl 2 V exp (2kg (±) - l^-in 

V W Ak(Ak 2 + l 2 )) \^m og n / 

<^E«xpL(i)-^Li(i-i) 



Z=2 

Consider the function 



sj k 3 log n 



, x 2o(x) 

^ x : = -n — wV, 2w > x G °' 1 

x(l — X) Z {1 + x z ) 1 

which is positive, convex and diverges to infinity as x — > or 1. Moreover for x £ 3/4] 
we have ^(x) < clog/c for some constant c > 0. Now, it is easy to see that clogk < -rt 
for k = e°( logn \ Thus we have 



3fc/2 / 2 \ 

^ 3/2 E-p(-g-^ 

Z=2 V 7 



\J k 3 log n 



< A ^ fc 3/2 exp(-</2 7 fc) 
1 -exp(-u 2 v /2 8 /c) 



as n — > oo as u 2 N /k « logn. 
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Now the function ip(x) < c'klogk for x £ [3/4,1 — l/2k] and c'klogk < u 2 N /Ak only 
when k < c log nj log log n for some small constant c > 0. In that case, we have 

2fe-l / / , N 9 ,/ , \ 2\ / -r r \ Z / 2fc 



2fc-l / 2 \ i 

< A; 3 / 2 V exp ( -^-(2k-l) 2 ) - - 1 — 
^ P V 2 fc / (& 3 logn) 3 / 8 



\/ A; 3 log n 



3k/2 

^3/8 ^ / u 2 \ 

as logn/fe, u 2 N /k 2 — > oo as n — )• oo. 

Now when A; > c log n/ log log n, we need to break the second sum into into two parts 
and take x n S> £;(log logn) 2 / logn. For x £ [3/4, 1 — log log n/ (2c log n)] we have < 
d log n < n^ / 4k and thus 

^ F y \ 9fr / 9/1 4- 9h\2\ Oh \ 



2k J 2(1 + (l/2k) 2 ) 2k \ 2k J I \^/Wk> 



2 G k 2K ' ) (Flogn) 3 / 8 



i=3fc/2 

2fc(l-log log n/(2c logn)) r , 

<^ 3/2 E ex P (-i(2*-Zr X 

3fc/2 

fc 3/8 e -3x„/4 ^ / ^ 

" (logn) 3 / 8 ^, 6XP V 2 6 k 2 

l=k log log n/c log n 
A ,l+3/8 e -3x„/4 / c 'ft(l gl ogn )2 

< — : '. ~ ~ — exp 1 



;n 



(logn) 3 / 8 \ logn 

as n — )• oo. For the last part we have 

2k-i / , , N 2 , / / n 2\ / \ '/2fc 

E -p Ms - A i - 



i=2fc(l-log log n/(2clogn)) 



2/t y 2(1 + (l/2k) 2 ) 2k \ 2k J Hypl^ 



;n 



< k 5 ^ 2 exp (2kg (1 -loglogn/2clogn))e- x " (1+o(1)) (^ 3 logn)- 1 / 2+ °( 1) 

< A; 5 / 2 exp (c'fc(log log n) 2 / log n - x n (l + o(l))) (A; 3 log n)" 1/2+o(1) -s> 
as n — > oo. Thus we are done. 



6. Proofs: Global optima 

6.1. Distributional results. We start by proving the comparison result, Lemma 3.2 using 
which we shall complete the proof Theorem 3.7. 

Proof of Lemma 3.2. Let Si be the matrix ((crij))fj=i where an = l,i = 1,2, . . . , N and 
Eo be the N x N identity matrix. For t = 0, 1 construct independent random variables 
X* ~ N(0, Et). For a general real number t £ [0, 1] let 

X* := v^X 1 + ^/T^X (6.1) 
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Note that 

X* = (X*,X*,...,X* r )~N(0,S t ) 

where = tSi + (1 — t)T,Q. 

Fix a smooth function G(x) of N variables x = (xi, x%, . . . , xn). Define Gjj(x) = 
dx dx ( x ) ^ or ^ — — Using the representation (6.1) and integration by parts one 
can easily prove that 

E(G(X 1 )) - E(G(X )) = VVy / E(G ij (X. t ))dt. 

"T' Jo 

We briefly sketch the proof for completeness. Note that X* = £^ 2 X°. Thus 

E^X 1 )) -E(G(X )) = / -f E(G(£ t 1/2 X°))cft 

Jo dt 

= / E(VG(S t 1/2 X°)^(S t 1/2 )X°)dt 
Jo dt 



^ 1 E(tr(|(S 4 1 / 2 )V 2 G(S t 1 / 2 X°)^ 2 ) ( it 
\ ^ 1 E(tr(|(E t )V 2 G(X*))^ 

J E(tr((Ei - £ )V 2 G(X*))dt = / E(G -(X*))dt. 

2 JO JO 



Fix e > 0. Take G e (x) = YliLi 1 ( M — x i)) m pl ace of G, where is the normal 
distribution function and <f> = We have 

lECG^X 1 )) -E(G £ (X°))| 

= /~2\ a ij\ ^{Pij(.u + sZ 1 ,u + sZ 2 ))dt 
Jo 

l<3 

where p\Ax,y) is the joint density of and Z±, Z 2 are i.i.d. Gaussian r.v.s. Taking 

e — > and noting that 



G £ (x) — >■ 1 < max Xi ^ u> , 

we have 

| P( max Xi < u) — P( max Z\ < u)\ < > \ ph(u,u)dt. 

l<i<N l<i<N 7 — f Jg 



Now 

p -v?/<l+a i:j t) 

Pij{u,u) 
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for ueE and thus 



p\Au,u)dt < 



-u 2 /(l+t) 



2ttV1 - t 2 

Changing the variable t to x = ^/ {\ — t) / {1 + t) we have 



dt 



pUu,u)dt < ■K- 1 e- u2/2 



< n~ x e 



l-u 2 /2 



-{ux) 2 /2 
1 + X 2 



-dx 



< 2*(u)|$(u) - $(0iju)\ 

< 2$(u)$(0^)min{l, |1 - B^u^O^u) mO^u)} 

< 2^u)$(9} j u)min{l,\l-9 ij \u(u + l)} 



where 0y = — a^) /(l + Q}j = niinj^j, 1} and we used the fact that <3?(x) is 
concave and <l(x) > 4>{x)/{l + x) for x > 0. This completes the proof. ■ 

Let us now prove asymptotics for the global maximum. 

Proof of Theorem 3.1: We start with the proof of part (a). Recall that \^ n (k)\ = N = (^) 2 - 
For any x G R, we have 

P(ajv(fcavg(W;s i *) — 6tv) < x) = P( max avg(W^) < £>at + a^x). 

Moreover, fcavg(W^) ~ N(0, 1) for all A G S^ n {k). Further for i.i.d. standard Gaussian 
r.v.s Z\, Z2, . . . , Zn-, standard results in extreme value theory (see Lemma 3.6) imply that 
for any x G R, 



P( max Zi < bw + a A }x) 

Ki<N N 



e~ e = P(-logT < x) as N 
where T ~ Exp(l). Thus it is enough to show that 



P( max /cavg(W\) < um) — P( max Zi < uat" 
\ey„{k) ' ' l<i<N 







as N — > 00 where un = b^ + a^x. First note that Cov(fe avg(WA), /cavg(Wy)) = stk 
where |A n A'| = (s, t), i.e., A, A' share s many rows and t many columns. 
Let un = bpj + a~j^x. Using Proposition 3.2 we need to show that 

-2 



-2 



N 2 $>(u N f 



£ 

l<s,t<k 
st^k 2 



k\ /k\ ( n— k\ (n — k 



k 



k - t 



k 2 + st 
k 2 -st 



stu 2 N /(k 2 +st) 







It is easy to check that N<&(un) —> e~ x as ./V — > 00. Lemma 5.6(a) completes the proof. 

Part (b) follows in an identical fashion using Slepian's lemma and Lemma 5.6(b). We 
omit the details. 

Now we move to the proof of part (c). For k fixed, from part (a) we have for any fixed 
x G R, 

P(a N (kM n (k) - b N ) > x) -> P(-logT > x) 
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as n — > oo where T is an exponential rate one random variable and N = (£) = \y n (k)\. 
Fix a subset A C E fcxfc and x G P. Define 

x n = (&tv + x/aN)/k. 

Given a matrix U, we will write U to denote the matrix U — avg(U)ll'. It is enough to 
show that for any Borel set A G ,6(P fexfc ), 

P(M n (fc) > x n ,W x * {k) - avg(W A . (fc) )ll' G A) 

- P(M n {k) > x n )P(W [fc]x[fc] G A) -> 

as n — > oo. We have by symmetry 

p :=P(M n (fc) > x n , W A . (fc) - avg(W A * (fe) )ll' G A) 
= J] P(A*(fc)= 7 ,avg(W 7 )>x ri ,W 7 GA) 

7G^n(A:) 

=iyp(A*(fc) = A,avg(W A ) > x n , W A G A) 

where A = [fe] x [A;] is the corner submatrix. Define S\ := {7 G y n {k) | 7 7^ A, A n 7 = 0}, 
namely the set of submatrices which have no overlap with A and ,yV\ = {7 G y n (k) | 7 7^ 
A, A n 7 / 0} so that the entire configuration space y n (k) = S\ U Now the event 

{A*(&:) = A} = {maxavg(W 7 ) < avg(W A ), max avg(W 7 ) < avg(W A )}. 
Moreover, max 7e ^ A avg(W 7 ), avg(W A ) and W A are independent. Thus 

n~ 1 p 

= P(maxavg(W 7 ) < avg(W A ), max avg(W 7 ) < avg(W A ), avg(W A ) > x n , W A G A) 
= P(maxavg(W 7 ) < avg(W A ), avg(W A ) > x„) P(W A G A) 

7S<?a 

— P(maxavg(W 7 ) < avg(W A ), max avg(W 7 ) > avg(W A ), avg(W A ) > x n , W A G A). 
In particular we have 

N^p- P(W A G ^l)P(maxavg(W 7 ) < avg(W A ), avg(W A ) > x n ) 
< P(max avg(W 7 ) > avg(W A ) > x n ). 

Since the upper bound does not depend on the set A, taking A = M kxk and simplifying 
we have 

P(M n (fc) > x n , W A * (fe) - avg(W A . (fc) )ll' G A) - P(M n (fc) > x n )P(W A G A) 
< 2iVP(max avg(W 7 ) > avg(W A ) > x n ) 

= 2iVP(avg(W A ) > x n ) ■ P(max avg(W 7 ) > avg(W A ) | avg(W A ) > x n ). 

Note that iVP(avg(W A ) > x n ) — > e~ x as n — > 00. Thus we have to show that 
P(max avg(W 7 ) > avg(W A ) | avg(W A ) > x n ) -)• 

76^A 
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as n -> oo. Let ^(s,£) = {7 G «5^(/e) | |An 7] = (s,t)}, 1 < s,i < fc. Clearly \<A\(s,t)\ = 
(1) (?) K) (fc-f) = 0(n 2fe - s -*). The union bound gives 

P(max avg(W 7 ) > avg(WA) I avg(WA) > x n ) 

< } P( max avg(W 7 ) > avg(WA) I avg(WA) > x n ). 

l<s,t<k,st^k z 

Note that Wa := Wa — avg(WA)ll' = {{w-ij)) is independent of avg(WA) and we have 
max i,je[fc] Wij = avg(W A ) + max iJe[k] Wij. 

For each 7 G -yV\{s, t), W 7 has exactly k 2 — st many elements outside the submatrix Wa 
and these entries are independent of the matrix Wa- Let F(W 7 ) be the average of the 
{k 2 — st many) entries in W 7 that are outside Wa- Thus max 7gi/ ^( s t ) avg(W 7 ) > avg(WA) 
implies that max 7e ^( s ^ F(W 7 ) > avg(WA) — (k 2 — st)~ 1 st maxj j e [ fc ] Wij. Thus 

P( max avg(W 7 ) > avg(WA) I avg(WA) > x n ) 

< P( max i ? (W 7 ) > x n — (k 2 — st)~ 1 st max wn) 

\&^x(s,t) i,je[k] 

Using Slepian's lemma we have 

P( max F(W 7 ) > x) < P(Vf^( a>t )i > x\Jk 2 - st) 

for all x G P where we use V n to denote the maximum of n many i.i.d. standard Gaussians. 
Thus 

P( max avg(W 7 ) > avg(WA) | avg(WA) > x n ) 

76-A(s,t) 



< P(Vi *( st )i > x n \/k 2 — st — {k 2 — st) 1 l 2 st max Wij) 

< \^ x {s,t)\P(Vi > x n Vk 2 -st - (k 2 - st)~^ 2 st max to y ). (6.2) 

i,j&[k] 

Now, \ J/x{s,t)\ = 0(n 2k - s ~ t ) and x n Vk 2 - st « y '{2k - 2st/k)2logn. Now it is easy 
to see that 2st/k < {s + t) 2 /2k < (1 - l/2fc)(s + t) < s + t - 1/k. Thus the probability in 
(6.2) converges to zero as n — > 00 and we are done. ■ 

6.2. Two point localization. Here we shall prove Theorem 3.4. Fix r > and recall the 
definition of k from (3.1) as well as k* . For any fixed to, let N n {m) denote the number of 
sub-matrices of size to with average greater than r, i.e., 

N n {m):= 1 {avg(W A ) > r} . 

Further note that if there is a sub-matrix of size to with average > r, then there exists a 
sub-matrix of size m — 1 with average greater than r. Thus the following Proposition 6.1 
completes the proof. ■ 

Proposition 6.1. For fixed r > we have the following asymptotics. 
(i) Let m = k* + 1. Then 

P{N n {m) > 0) as n -)• 00. 
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(ii) Let m = k* — 1. Then 

F(N n (m) > 0) —?■ 1 as n -> oo. 
Proof of Proposition 6.1. Define 

/ n (x) := Q #(xr) 

for a; G [l,n]. It is easy to see that E(A^ n (m)) = f n (m)- Moreover, using Lemma 5.1 
and Stirling's approximation T(x + 1) = \f2iix x+l /' 2 e^ x+0 ^ l ^ x " > for x > 1, for any constant 
cGRwe have (with x = k) 

fn(x + C) 



/n(a:) 



r(x + i)r(n - x + 1) \ 2 $(0 + c)t) 



r(x + c+ l)r(n - x - c+ I) J $(xt) 



rr 2x+2(„ „,\2n-2:r+l „ „ 

x [n- X) _ e -(2ca'+c 2 )r 2 /2+0(l/x) 



(x + c) 2 *+ 2c + 2 (ra - x - c) 2 "- 2 ^" 2 ^ 1 



(l-x/n) 2c /"' ( rr'V ( c 2-OU/>.) 



(1 + C /x) 2 *'+ 2c + 2 (l - C /(n - x ))2n-2x-2 C +l V X 

Using (3.2) one can easily check that 

^ exp(— kr 2 /4) — > 1 as n — > oo. 

Thus 



or oo as n — > oo 

fn(k) 

depending on whether c > or c < 0. In particular, we have 

E(iV n (/c* + 1)) -> and E(iV n (/c* - 1)) -> oo as n -> oo. 

Part (i) now follows easily from the fact that P(iV n (£;* + 1) > 0) < E(iV n (£;* + 1)). To 
prove (ii), we shall use the second moment method. To simplify notation, for the rest of 
this proof we write k = k* — 1. We have already proved that E(A r ra (/c)) — > oo as n — > oo. 
By the second moment method, it is enough to show that 

E(iV n (fc) 2 ) 

;r—r; TTTTTi — > 1 as n — > oo. 
(EiV n (A;)) 2 

Note that the collection of random variables (avg(W A ) : A G ^ n (k)) is transitive, in the 
sense that for any Ao,Ai G r y n (k), there exists a permutation 7r : J^ n (£;) — > y n {k) with 
7r(Ao) = Ai such that 

(avg(W A ) : A G & n {k)) = (avg(W 7r(A) ) : A G ^ n (fe)). 

A simple calculation using this transitivity now implies that to prove the second assertion, 
it is enough to show for a fixed Ao G y n (k), 

T _ ST F(avg(W A )>r|avg(W Ao )>r) n 

n '~ ^ E(NJk)) U [b ' 6) 
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as n — > oo. Note that the vector (k &vg(W \) , k &vg(W \ Q )) has a bivariate normal distri- 
bution with variance one and correlation st/k 2 where s is the number of common rows 
between A, Ao and t is the number of common columns between A, Ao- 
Define, 

„, _ fn\ ~ 2 fk\ (n - k\ (k\ (n - k\ F(Z st > kr \ Z > kr) 
E[S ' ty -{k) [ s )[k-s)[t)[k-t) ¥(kr) 

for 1 < s,t < k where Z st = k~ 2 stZ + y/l - k~ A s 2 t 2 Z' and Z,Z' are i.i.d. standard 
Gaussian r.v.s. Thus we have 

k k 

8=1 t=l 

Clearly E(k,k) = l/E(iV n (fc)) — > as n — > oo. We need to estimate E(s,t) for st < 
k(k-l). 

Using Lemma 5.4 and Lemma 5.1 with 6 s t := y fjrffj we have 



T(Z st > kr \ Z> kr) 2$((9 s4 /ct) Ik 2 - st ( rck 2 T 2 



$(fcr) ~ *(fcr) ~ \k 2 + st K \k 2 + 



rc 



for st < k(k — l) and kr > 1. Now we use Lemma 5.6 with N = (?) , x = (k — k)raN ~ /cr 2 . 
Thus we have I n — )• as n — )• oo and we are done. ■ 

7. Proofs: Local optima 

7.1. Proof of Structure Theorem 3.7. Let 7?.^ and Cfc be the events that the sub-matrix 
C = Wm x u.i is row optimal and is column optimal, respectively. Clearly P(7£fc) = P(Cfc) = 

(?) 1 and we need the probability of the event I k := n as well as the conditional 
distribution of C given . 

Let Cj. = A; -1 X^=i w ij> c -j = k -1 Ya=i w ij-, c - = k~ 2 Yli,j=i w ij for 1 < < We use 
the ANOVA decomposition of the Gaussian matrix C which gives that 

Cij = Cij + (cj. - c.) + (c.j - c.) + c. 

where c"ij = — Cj. — c.j + c... Under the gaussian assumption the random variables 
C = ((cjj))ii=i) ( c «- ~~ c ~)i=ii ( c -j ~ c --)j=n c - are independent and obviously independent 
of the remaining n — k row and column averages (Q.)™ =fc+1 , (c.j)™ =fc+1 . 
Clearly we have 

IZk := { min c.j > max c.j} = {c. — max {a. — c.j} > max c.j} 

1 < J <^ k<j<n k<j<n 



and hence 



Cfc := { min Cj. > max a.} = {c. — max {a. — c«.} > max c«.} 

Ki<fc k<i<n Ki<k k<i<n 



X k = lZ k C\C k = {c. > max {c.j, Cj.}} 

fc<j<n 



n { max {c. — c.j} < c. — max c.j} 

1 < j < k<j<n 

n { max {a. — Cj.} < a. — max q.}. 

l<i<k k<i<n 
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Define M n - k = a n (Vkmax k<j < n c.j - b n ),M' n _ k = a n (Vk m&x k<i < n q. - b n ) for the re- 
centered and rescaled maxima of the remaining row and column averages. From Lemma 3.6, 
(M n - k , M^_ k ) converges in distribution to (— logT, — logT') where T, T' are i.i.d. Expo- 
nential rate one random variables. Now the event X k can be expressed as 

I k = {Vkc. >b n + a^ 1 max{M n _ fc , M' n _ k }} 

n {Vk max fa. — c.A < vkc — (b n + a~ l M n - k )\ 

l<j<k 

n {Vk max {a. - cj,} < Vkc. - (b n + a^M^_ k )}. (7.1) 

l<i<k 

Note that kc, Vkci., Vkc.j are standard Gaussian random variables. Let Q(x) = P(Z > x) 
be the tail probability of standard Gaussian distribution. Define F k {x) = P(maxi<j</%(Z — 
Zi) < x) where Z\, Z-i, . ■ ■ , Z k are i.i.d. standard Gaussian r.v.s with Z = k~ l X^i=i 

Using the independence resulting from the ANOVA Decomposition and using (7.1) gives 

P(X fc ) = E(F fc (fc-V 2 Z -b n - a~ x M n „ k ) (7.2) 
F k {k- l / 2 Z-b n -a- l M' n _ k ) (7.3) 
\{k- l ' 2 Z >b n + a~ l max{M n _ fc , M' n _ k }}) . (7.4) 

The idea of the rest of the proof is as follows. We will show that the event (7.4) has 
probability of the order (logn)( fc_1 )/ 2 ra~ fc and conditionally on this event (a n (k~^' 2 Z — 
b n ), M n - k , M' n _ k ) converges in distribution. In particular k~ x l 2 Z — b n — M n - k /a n = 
@p(a~ 1 ). Using Lemma 5.3(a) we get that the first two random variables namely (7.2) 
and (7.3) behaves like a n ^ k ^ ~ (logn)~^ fc_1 ^ 2 . Combining everything we will finally get 
the order of the probability to be (log 

Using Lemma 5.2 with 9 = k we have that 

Vk{V^b n ) l - k n k F(k~ 1/2 Z >b n + a~ l x) -»• e~ kx 

as n — > oo uniformly over \x\ <C a n . Using the fact that m&x{M n - k , M^_ k } converges in 
distribution to — log(T/2) where T ~ Exp(l) we have 

Vk(^b n ) X ~ k n k F(k~^ 2 Z >b n + a" 1 max{M„_ fc , M' n _ k }) 
-> 2" fc E(T fc ) = 2~ k k\. 

Now for any x, x' , y < min{x, x'} fixed, we have 

Vk(^b n f- k n k V(k- l l 2 Z >b n - a-Hogy^n-k < -logx,M;_ fc < - log a/) 
— > y k e~ x ~ x as n — > oo. 

From here it easily follows that, conditional on {k~ x l 2 Z > b n + o^ 1 max{M n _j., M'u}}, 

{a n (k- l ' 2 Z -b n ),M n - k ,M n _ k ) => (-l og G,-log(G + y),-log(G + y / )) (7-5) 

where Y,Y' are i.i.d. Exp(l) and G ~ Gamma(fc, 2) with density , k 2 ^ l y X k ~ 1 e~ 2x , x > 0. 
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Using Lemma 5.3(a) we finally have 



11 



lim ( ;)a n ' £ - 1 P(X fc ) 



2 -* (27r )(*-i)/2 



'k 



lim E(a n 2 ( fc - 1 )F fc (a,; 1 log(l + Y/G))F k { a - 1 log(l + Y'/G))) 



_ (2vr)( fc - 1 )/ 2 /| 
In particular, we have 



E((log(l + Y/G) log(l + Y'/G)f- 1 ) 
k 



P(2*) 



where 



(™)(logn)(fc-D/ 

£,2fc+l/2 



— + o(l)) asn->oo 



E((log(l + Y/G) log(l + y'/G))"- 1 ) 



(7.6) 



7r (fc-l)/2 2 2fc-l^|2 

The above gives the asymptotic probability of C being a local optima. Let us now find 
the asymptotic conditional distribution of the matrix itself given X k . In matrix form, the 
ANOVA decomposition can be written as 



(7.7) 





Cl. 


— c. . 




C.l 


— c. . 


C = c..ll' + C + 


C2. 


— c. . 


l' + l 


C.2 


— c. . 




Ck. 


— c. . 






— c. . 



where C = (c^) where 5jj = Cy — Cj. — c.j + c, 1 < i, j < k is independent of the event 
If.. This immediately gives the second term Z in the structure theorem. Now note that 
by (7.1) on X k the row sums satisfy 

Vk max fa. — a,} < VXa. — (b n + a~ l M n _u) 
i<i<fc 

Here the term on the left has distribution maxi^^ {Z — where the Zi are i.i.d. stan- 
dard gaussian random variables and Z = avg({Zj} ls , is , fc ). Further by the ANOVA de- 
composition, this random variable is independent of the term on the right which by 
(7.5) is of order Bp(a~ 1 ). In fact, (7.5) implies that conditional on the event {k l l 2 c. > 
bn + a-n 1 max {^n-fc) M' n _ k }}, the random variable a n {ykc. — {b n + a~ l M n _k)) converges in 
distribution to log(l+ Y/G). Thus for the third term in the ANOVA decomposition in (7.7), 
on the event X k intuitively one is looking at the distribution of [Z\ — Z, Z2 — Z, . . . , Z k — Z) 
conditional on maxi^^/j \Z — Z{\ ^ Op(a~ 1 ) which is exactly the type of event Lemma 
5.3(b) is geared to tackle. An identical argument applies to the last term in (7.7). 

More precisely, using Lemma 5.3(b), (7.5) and the form of X\~ in (7.1), a simple 
conditioning argument shows that the conditional distribution of a n (c. — b n /\fk,c\. — 
c, . . . , Cfc. — c, c.i — c, c.fc — c.) converges in distribution to k~ x l 2 {— log G, (kU\ — 1) log(l + 
T/G),...,(kU k - l)log(l + T/G),(kU[ - l)log(l + T'/G),(kU' k - l)log(l + T'/G)) 
where (U\, U2, ■ ■ ■ , E/&), (U[, U' 2 , ■ ■ ■ , U' k ) are i.i.d. from Dirichlet(l, 1, . . . , 1) distribution and 
(G, T, T) has joint density 

g,t,t'>Q. 



oc(log(l + t/ ff )log(l + i7 ff )) fe -y- 



-l e -t-f-2g 
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The (log(l + t/g) log(l + f / ' g)) k ~ l term is arising from the {k — l)-dimensional volume of 
the simplexes {maxi<j<fc{x — Xj} < \og(l + t/g)} and {maxi<j<fc{x — Xj} < log(l + t'/g)}. 
U 

7.2. Variance asymptotics. The previous section gave amongst other things, asymp- 
totics for the expected number of local optima E(L n (/c)). The aim of this Section is to 
prove Theorem 3.9, giving asymptotics for variance of L n (k). The first step is to under- 
stand the joint distribution of two matrices being locally optimal, which turns out to have 
a highly non-trivial structure. We start with a proof of Lemma 3.11 which bounds the 
probability for two overlapping matrices being locally optimal. 

7.2.1. Proof of Lemma 3.11. Recall that this Lemma gives asymptotic bounds for the 
probability of the event B s i & (see Figure 3.1) for two matrices having k — s rows and k — t 
columns in common to be both locally optimal. We define the following matrices 

X l = W [s] x [t] j X 2 = W[ s ] x [ t+1)fc ] , 

X 3 = W[ s+1)fe ] x [f], X 4 = W[ s+1)A .] x [ t+1)A .], X 5 = W[ a+ i jfc ] x [ fc+ i | fc +t ] 

X 6 = ~W[k+l,k+8]x[t+X,k]j = W [k+l,k+s]x[k+l,k+t]- 

Let Si = avg(Xj) and 0j = number of entries in Xj for i = 1,2, ... ,7. In particular we 
have 

e 1 = e 7 = st, e 2 = 6 = s(k-t), 

e 3 = 6 5 = (k-s)t and 6 4 = [k - s)(k -t). 
Clearly the joint density of (Si, 1 < i < 7) is given by 

7 

11 v/0i/27rexp(-0 iS ?/2) for ( Sl ,s 2 , . . . , s 7 ) G E 7 . 



i=l 



Define 



k k 
M c = max > Wi j and Ml = max y w s +i , 

k+t+l<j<n ' J k+t+l<j<n^-^ ' J 

i=l i=l 

i.e., M c is the maximum column sum of the sub-matrix W[Wx[jfc+t-fi,n] an d M' c is the 
maximum column sum of the sub-matrix Wl + i |S ^u x [fc+t+i )n i • Similarly define 

k k 
M r = max > Wi j and ML = max > Wt+i j 

k+s+l<i<n ^— ' ' J fc+s+l<i<n ^— ' ,J 

as the maximum row sum of the sub-matrix ^W[k+ s +i,n]x[k] an d the maximum row sum of 
the sub-matrix Wrfc +a+ i n i x r t+ i t+ w 3 respectively. For a real number x S R, let V(x) be 
the set 

2?(a;) = {{s\, S2, ■ ■ ■ , S7) S R 7 I ts\ + (k — t)s2 > x, ssi + (k — s)s 3 > x, 

SS2 + (k — 5)54 > x, is3 + (A; — t)s4 > x, 
(k — t)s4 + ts$ > x, (k — s)s4 + SSq > X, 
(k — s)ss + ss~/ > x, (k — t)sQ + tsj > x}. 
Note that T>(x) is decreasing in x. It is easy to see that 

B s , t ,k C {(5i, 5 2 , • • • , SY) G r>(min{M r , Af^, M c , M' c })}. 
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Now (M r , M' r , M c , M' c ) is independent of (S h 1 < i < 7) and M r = M' r , M c = M' c . Thus we 
have 

P(B 8)tlfc ) < 2E(/(M r ) + /(M c )) 
where /(x) := P((5i, 5 2 , . . . , S 7 ) G D(x)). We claim that 

for x > 0. Using standard calculus one can easily check that under T>(x), Yll=i ^i s f ls 
minimized at Sj = ctj, 1 < i < 7 where 

(3k- s- t)x (2k - t)x 

2k 2_ st > a2 " a6 -^3^' 

(2/c — s)x 2kx 

03 = 05 = TT?9 ano - °4 ~~ 



2fc 2 -st * 2k 2 -sf 

We will not use this fact in the subsequent calculations, however it will help us to estimate 
P(V(x)). Note that the above vector (a±, 02, . . . , 07) makes all the inequalities in V(x) 
equalities. We have 

f(x) :=F((S 1 ,S 2 ,...,S 7 )€V(x)) 
7 

T\v / 6 i /27rexp(-6 l s 2 /2)ds i 
7 

TT v / ^/2vrexp(-^(s i + a;) 2 /2)dsi 
*>(o) f=i 

7 7 
= TT V^/27rexp(-^af/2) / exp(- V ^(s 2 /2 + a^))d Si . 
ti Mo) U 

Further note that 

2k 2 - st 7 

^i a i s i =s(2k — S — t)(tS\ + (k — t).S2 + (k — t).SQ + ts 7 ) 

x i=i 

+ H(ssi + (A; — 5)53 + (At - 5)55 + ss 7 ) 

+ s(/c — t)(ss2 + (A; — s)«4 + (A; — s)s^ + ssg) 

+ (k — s)(k — t)(ts 3 + (k- t)s 4 + (k- t)s 4 + ts 5 ) 

which is non-negative under V(0). Thus we have 

7 7 
/(*) < TT V / ^exp(-^a 2 /2) / exp(- V t s 2 i /2)ds i 

7 

<exp(-^a 2 /2). 
i=l 

Simplifying we have 

1=1 v 
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This proves the claim (7.8). Note that M r — ^fkV n -k-s and M c = y/kV n -k— t where 
V n = max{Zx, Z%, ■ ■ ■ , Z n } and Z^s are i.i.d. N(0, 1). Thus to complete the proof it is 
enough to show that for any constant 9 > we have 

E(exp(-#max{T4,0} 2 )) < rj(d) exp(-6l%) 

for some constant rj(9) > where b n satisfies e~ bn ^ 2 = \[2/Kb n jn. Using 9 = (1 — (k — 
s)(k — t)/(2k 2 — si)) and the fact that b n = y/2\ogn — log (Att log n)/y/S log n gives the 
bound asserted for 1P(B S ^ The following Lemma 7.1 completes the proof. ■ 

Lemma 7.1. Let V n := max{Zi, Z%, . . . , Z n } where Z^ 's are i.i.d. N(0, 1). For any constant 
9 > we have 

E(exp(-#max{T4,0} 2 )) < rj(6) exp(-06 2 ) 
for some constant n{9) > for all n > 1 where b n = \/2 log n — log(47r log n) /1/8 log n. 
Proof of Lemma 7.1. Define X n = b n (b n — V n ). We have 

E(exp(-#max{K,0} 2 )) < 2~ n + exp(-#& 2 ) E(exp(20X n )l{K > 0}). 

It is easy to see that 2~ n exp(#6 2 ) is bounded for any n. Thus we have to show that 

E(exp(2#X n )l{V„ > 0}) is uniformly bounded in n. By Lemma 3.6, we have X n =>- logT 
where T ~ exp(l) and E(T 2e ) < 00. Thus E(exp(#X n )l{X n < c}) is uniformly bounded 
over n for any fixed c > 0. Moreover V n > implies X n < 6 2 . Thus we need to bound 

E(exp(#X n )l{c < X n < 6 2 }) 

for an appropriate choice of c. We will break the interval [c, 6 2 ] into several subintervals 
and estimate the contribution from each subintervals. Note that 

P(X n > x) = (1 - ${b n - x/b n )) n < exp(-n$(6 n - x/b n )). 

Moreover, we have for all x £ [0, b n ] 

b n (b n -x/b n ) x 2 /2b 2 
l + {b n -x/b n y 

for some constant C € (0, 26/e]. Thus using the bound > u 2 /(^/2tt(1 + u 2 ))e~" 2//2 

from Lemma 5.1 and the fact that ne~ b ™/ 2 / (V2irb n ) = 1 + o(l) we have 

P(^« > a?) < exp(-Ce a; ) 

for all x £ [0,b n ]. 

We take x = b 2 n and x i+1 = \og{26xi/C) for i > 0. Note that 29/C > e. Let c* > 1 
be the largest solution to the equation x = log(29x/C). It is easy to see that 
i — > oo. Thus there exists such that c* < < 2c* < x^. Note that > Cxf +1 /4# 
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implies that Xi > A6{Cxk+\j '46) 2k+1 % /C. Thus we have 

k 

E(exp(#X n )l{2c* < X n < b 2 J) < ^E(exp(0X n )l{x m < X n < x t }) 

i=0 

k 

<J^exp(^)P(X n >x i+1 ) 

i=0 

k k 

< ^exp(fei - Ce x > +1 ) < ^exp(-^Xi) = O(l). 

i=0 i=0 

This completes the proof. ■ 

7.2.2. Proof of Variance asymptotics: Theorem 3.9. Let us now complete the proof of the 
main result. 

Proof. Let p n = P(Wm x [fc] is locally optimal as a sub matrix of Wr n ] x r n |). By Theo- 
rem 3.7 we have p n = (1 + o(l))^fc/((^) (logn)^ fc_1 ^ 2 ). By symmetry we have 

Var(L n (fc)) 

= Cov(1{Wa is locally optimal}, 1{W 7 is locally optimal}) 

\,~/&y„(k) 

= (f) EE Q) (*) ( n ~ k )( n ~ t k ) Cov(1{w w x w is locally °p timal i< 

l{W [s+1 tS+k ]x[t+i,t+k] is locally optimal}). 

Define 

v n (s,t) := (™) rj Qj ^ ^ ^ ^ J Cov(l{W[ fe ] x [ fc ] is locally optimal}, 

l{Wr s+ 

i,s+k]x[t+i,t+k] is locally optimal}) (7-9) 

for < s,t < k. We will show the dominant contribution comes from the s = t = k 
case, i.e. when the matrices are completely disjoint, sharing no rows and columns and in 
this case v n (k,k) ~ n 2k = n 2fe-2+2/(fc+i) logarithmic corrections. We consider 

several different cases depending on the values of s, t. 

Case 1. s = t = : The matrices are the same. Clearly here < v n (0, 0) = 

(fc) 2 pn(l-Pn) = 0((logn)-(^ 1 )/ 2 ). 

Case 2. s = 0,t > or s > 0, t = 0: Here the matrices have identical row or 
column sets but do not overlap. In this case obviously both matrices cannot be simul- 
taneously locally optimal. Thus the covariance term is — p\ and the contribution is 

|t; ft (a,t)| = 0{n 2k+s+t p 2 n ) = 0(n k (log n) 1 ~ k ). 
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Case 3. < s,t < k: Using Lemma 3.11 we have 

< Cov(l{W [fc]x[A .] is locally optimal}, t{W [s+hs+k]x[t+ht+k] is locally optimal}) 
< r,(s, t, fc)(logn/n 2 )^ fc ( fc - s )( fc - t )/( 2fc2 ' st ). 
Thus we have 

< v n (s,t) = 0(n s+ * +2fc( ^ s)(fc "* )/(2fc2 " s * ) (logn) fc ~ fe(fe " s)(fc "* )/(2fc2 " s * ) ) 

Note that 

2k(k - s)(k - t) _ 2k , 2k 



2k 2 - St k(3k-s-t) _ -, - k(3k-s-t) _ , ■ 

(k-s)(k-t) 1 (fc-(s+t)/2) 2 1 

Thus defining 6 := (s + t)/2k we have 

2k(k-s)(k-t) , / (l-#) 2 , 
s + t + _ n A - < 2fc ( + ^ 4r = 2fc 



2P-st ~ \ 2-8 2 } \2-9 2 

Now S?l^f^ = 6 %-9^' > ^ implies that |5§7 is a strictly increasing function of 6. 
Also note that 9 G [1/fc, 1 — 1/fc]. Thus we have < v n (s,t) < v n (k — 1, k — 1) for all 
s,t6 {1, 2, . . . , A; — 1}. Now for s = t = k — lwe have 

2k(k - s)(k -t) 2k 
s + t+ „ in n = 2k-2 + 



2k 2 -st k 2 + 2k-l 

2k 2 2{k-l) 



k + 1 (k + l)(k 2 + 2k-l)' 

Thus for < s, t < k we have 

2fc 2 2(fe-l) , fc 

< v n (s,t) = 0(n k+1 (HiJiiW^gn)" 1 P+5fc=T). 

Case 4. s = t = k: Here we will show that 

„ n (Jfe, fc) = + (l))n 2fe2 /( fc+1 )(logn)- fe2 /( fc+1 ) 

for some constant > 0. Let X k ^ n _ k be the event that Wny x [jfc] is locally optimal as a 
submatrix oiW [k]u[2k+ljn]x[k]u[2k+ltn] and let T' k , n _ k be the event that W[ fc+1]2fe ] x[fc+1;2fc ] is 
locally optimal as a submatrix of Wu;+i )n ]x[fc+i,n]- Clearly these two events are indepen- 
dent and P(X fci „_ fc ) = ^(Z'k,n-k) = Pn-k- Let W* = (t/^fcxfc, W** = (w**) kxk denote the 
matrices W[ fc ] x [ fe ], W[ fc+li2 &]x[fc+i,2fc] respectively conditional onI^ n _ fc nI^_ fc . From The- 
orem 3.7 we know refined asymptotics for this distribution. Write C := 'W[k]x[k+l,2k]^ C := 
Wr^ + i j2 fc]x[ifc] f° r the remaining two submatrices that determine the local optimality of 
w [k]x[k], w [fc+i,2fc]x[fc+i,2fc]- See figure 7.2.2 for a pictorial description. 
By conditioning first on I n -k H we have 

P(W [fc]x[fc ] & W [fc+lj2fc ] x[fc+1)2fe ] are locally optimal) 

= Pn-k E^l{minio*. > maxq.,min w* > maxc.j}l{minu)*.* > maxQ., minu>** > maxc! 3 }j 

Now by the ANOVA decomposition, note that maxj Cj. — c, maxj c.j — c..,maxjC^. — 
c[. , maxj c'.j — c'.,c..,c!. are independent. Let d..,d[. be i.i.d. copies of c..,c'.. Then 
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W[t]x[*] 


c 


W [fc]x[2fc+l,n] 


c 






W[ 2t+ i,„] x [fc] 







Figure 7.1. The event Ik,n-k corresponds to the matrix Wrj.i x ru being 
optimal in the light gray region and the event I' k n _ k corresponds to the 
matrix *W [k+i,2k]x[k+i,2k] being optimal in the dark gray region. 



(maxcj,, maxc.j) is independent of (d.. + max a. — c, max c[j — c[. +d[.) = (max a. , max c' 
This implies 

Pn = Pn-k minw* > max q. , min w* > maxc.j} 

l{minw** > maxcj. — c. + cL,minw** > maxc'^ — c{. + d'.})). 

Putting these equations together results in 

Cov(l{W[ fe ] x[fc] is locally optimal}, l{W [k+h2k ]x[k+i,2k] is locally optimal}) 
= Pn-k lE^ljminu;*. > max q., minw* > maxc.j} 

• ^l{minw** > max a. , min w** > maxc'j} 

— l{minw** > maxcj. — c. + d.,minw** > maxc^ — c[. + d[.} 

Define the random variables 

E : = minw* — max(q. — c[.), 
G : = minw** — max(cj. — c), 



F := minw* — max(c.j — c), 
H := minw** — max(c'- — c'.). 
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Note that the random variables E,F,G,H are independent of c..,c[.,d..,d'... Using the 
fact that by construction c. = d.. and c[. = d'„, we have 

Pn 2 fc Cov ( 1 { W [fe]x[fc] is locally optimal}, l{W[ k+lt2 k]x[k+l,2k] is locally optimal}) 
= E(P(c. < min{F,G})P(c'. < mm{E,H}) 

- P(c. < F) P(c. < G) P(c'. < E) P(c'. <H)\E, F, G, H) 
= E(P(c. < min{F,G})P(c'. < mm{E,H}) 

(1 - P(c. < max{F, G}) P(c'. < max{£, #})) | E, F, G, H) 

= EMP(c < min{F,G})P(c'. < min{E,H}) 

P(c. > max{F, G}) + P(c. < max{F, G}) P(c'. > max{E, H})^j \ E, F, G, H^j . 

Note that from the Structure Theorem 3.7 we have that a n (w** —b n j vk) , a n (w*j—b n /Vk) 
are tight. Thus we have 

Pn 2 /k Cov ( 1 { W Wx[fc] is locally optimal}, l{W[ fe+ i )2 fc] x [fc+l,2fc] is locally optimal}) 
= (2 + o(l)) P(c > max{F, G}) = (2 + o(l)) P(maxc.j > min w*j , max a. > minu;**). 

Using Lemma 3.10 with s = t = k, 9 = 1/ \fk we have 

v n (k, k) = {t \ ( ^ 1 Pn-k(2 + P(maxc.j > min to*-, maxcj. > min-tt)**) 

= e( n 2fc-2fc/(fc+i) (logn)fc /(fe+i)-i-(fc-i) ) = (( n /yio^)2fc 2 /(fc+i)). 

Case 5. s < k and t = k: In this case note that 

p ( w [fc]x[fc] is locally optimal, W [s+l s+fc]x[fc+12fc ] is locally optimal) 

< p ( w [fc]x[/c] is row optimal, W [s+liS+fc ] x [ fc+lj2fe ] is row optimal) 



, 2 

n 



k 

Thus we have \v n (s,k)\ = 0(n 2k+s+k ~ 2h ) = 0(n 2k ~ 2 ) for s < k — 2. We need to consider 
the case t = k, s = k — 1 separately as 2k — 1 > 2k 2 /{k + 1). However, using a similar 
analysis done in case 4 and the fact that P(maxcj. > max it?**) = 0(y / Iogn/ri) where 
C = W[ fe _i] x [ fc+1)2 fc] we have 

\v n {k - l,k)\ = 0(n 2fc+2fc - 1 - 2fe - 1 7ioi^) = 0(n 2k - 2 ^/k^l). 

Note that in the case when s = k — l,t = k, the number of sub matrix pairs and covariance 
term balance each other in a subtle way. 

Case 6. s = k and t < k: Similar to Case 5. 

Combining everything we finally have 

Var(L n (A:)) = (u k + o(l))(n/ 'y^) 2k ' 
for some constant v k > where the o(l) term decays like 



(log n / n 2 )( k + 1 )( k ' 2 + 2k - 1 ') (log n) 



2k-l 
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8. Proof of the Central limit theorem 

The last section analyzed first and second order properties of the number of local optima 
L n {k). The aim of this section is to prove the Central Limit Theorem 3.12 for L n (k), for 
fixed k ^ 2. For submatrix A = I x J £ define 

X\ := 1{W\ is locally optimal for W[ n ] x [ n ]}. 

Write L := L n (k) = Ylxe^nik)^ ^ or * ne total number of locally optimal sub matrices of 
size k x k. To emphasize the dependence on the underlying matrix W := Wu^u, when 
necessary we will write X A (W), L(W) instead of I\, L respectively. 
Let 

p n = E(X A ), ii = E(L) = Q p n and a 2 = Var(L). 

From Theorem 3.8 and Theorem 3.9 we have 

9 k n k f jt n 2fc2 /( fc+1 ) 
" = fc!(logn)(^)/ 2 (1 + ° (1))andCJ = (lo g nr/(^) {1 + 0{1)) 

for some constant 6k, > 0. Thus 

I = (1 + o(1)) ^/(^D(lo a gn)V( 2fc+2 ) = f 8 " 1 ) 

where otk = k\vk/&k > 0. Let W = ((w'^)) be an i.i.d. copy of the underlying matrix W. 
For any fixed submatrix A = I x J e J^ n (&), define 

A J "^afe ^ either a G / or 6 G J 
a I u> a 6 if a ^ I and b ^ J, 

W A = ((itfy)) ana ^ ^ A := -^(W A ). Thus we replace all n entries for the row set and column 
set of A by independent and identical entries w^ b . If A is chosen uniformly at random 
from 5t* n (k), it is easy to see that W A and W form an exchangeable pair. However we 
will not use the exchangeable pair approach for Stein's method as the conditional error 
E(L A — L | W) is not linear with L. Recall from the discussion on Stein's method in 
Section 4.4, in order to prove that L = (L — fi)/a, one needs to bound | ~E(g'(L) — Lg(L))\ 
for g in the class of functions T>' in (4.1). We will use a direct argument to bound this 
quantity. 

First note that Z A (W) is independent of L x . Thus for any twice differentiable function 
/, we have 

E((L - M )/(L)) = Y, mxf(L) - pnf(L)) 

A 

= J>(X A (/(L) - /(L A ))) 

A 

= ]T E(X A ((L - L x )f'(L) -\{L- L A ) 2 /"(^))) 
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A 



where is a random variable. In particular with L = (L — [i)ja and fix) = g((x — //) /a) 
we have 



| E(Lg(L) - g'{L))\ < «^ E|J> A E(L - L x \ W)-a 2 \ + ^ E J> A (L - L A ) 2 . 

^ A ° A 

Note that by symmetry 

E^X A (L - l/) 2 = fiE((L - L A °) 2 | X Ao ) (8.2) 



where Ao = [k] x [k] and for simplicity we write E{- \ Z A() ) := E(- \ Z\ = 1). Thus using 
Lemma 4.1 we have 

dw(L,N(0,l)) < ^E|^X A E(L-L A | W) - a 2 \ + E((L - Z>) 2 | J Ao ). (8.3) 



A 



Recall that, for A, 7 G 5? n {k), |A n 7) = (s,t) implies that A and 7 share s many rows 
and t many columns. For fixed A G y n (k), define 

y x (s,t) := {7 G y n (fc) I |AD7| = (k — s,k — t)}, < s,t < k. 

Thus ^ A (s, t) consists of the set of submatrices which s rows and t columns different from 
A. Write 

S x (s,t):= Yl (£y"£y(W A )) 
so that we have L — L x = ^o<s t<k &x( s i Clearly 

i^i=0(J)(";*)( B 7*)=°^)- 

Let 

Un(a,t) :=E(S A (s,t) |Z A ) 

By symmetry, this term is the same for all A. Recall from (7.9) that the variance of L n (k) 
could be expressed as a 2 = t v n (s, t) where 

n\ 2 fk\ fk\ (n — k\ fn — k 



v n (s,t) : = ( k j y t j[ s J f J Cov(l{W [fe]x[fc] is locally optimal}, 

1{W [S+ 

i,s+k]x[t+i,t+k] 1S locally optimal}) 
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A simple conditioning argument shows that v n (s,t) = fiu n (s,t). Now let us consider the 
first term in the bound (8.3). 

E|^X A E(L-L A I W) -a 2 \ 
x 

k k 

^EE E I E Z\HS\(s,t) | W) - fiu n (s,t)\ 
s =o t=o \&y n (k) 

k k 

< • E I J a nSx (s, t) - u n (s, t) I W)| + \u n (s, t)\--E\L-n\) 

s=0 t=0 

k k k k 

<^5>(|E(S Ao ( S ,i)| W)-u n (s,t)\ |X Ao ) + a^^K( S) i)|. 

s=0 t=0 s=Q t=0 

Similarly for the second term in (8.3) we have 
k k 



e((l - z>)2 1 x Ao) < \jnsx {s,ty | X Ao 



s=0 i=0 
k k 



< EE(M s >*)i + v Var (^o(^*) i 2a ))- 

s=0 i=0 

The proof of the variance estimate in Theorem 3.9 shows that u n (s,t) > for si > 
and u n (s, t) = — \5f\ (s, i)|p n for si = 0, s + i > 0. In particular we have 

k k ^ k k 2 

£ £ k( S , oi = -EE *)i ^ v 

s=0 t=0 ^ s=0 t=0 ^ 

for some constant c > 0. Combining, the bound (8.3) reduces to 

k k 

d w (L, N(0, 1)) < E E ^ E d E ( 5 ^ («, *) I W) - u n (s, t)\ | X Ao ) + — 

s=0 t=0 G ^ 
k k 



+ (\/^ + EE\/|s Var ^o(M)l^o) • (8.4) 

\ V ^ s =0 t=0 V / 

From (8.1) it follows that o j [i — > as n — > oo. Moreover, for st = we have (S^s, t)\ < 
1 a.s. Note that 

2 3 
°L _ n fe-2+2/(fc+l)+o(l) and °_ _ n 2fc-3+3/(fc+l)+o(l)^ 

Thus the case si = is negligible and we are left to prove that 

k k 

a 2 



F i : = E E ^ E d E (^o (*, *) I W) - u„(s, t)| | X Af) ) -> 



5 = 1 t=l 

k k 



8=1 t=l 



P 2 : = E E a/4 Var(5 Ao (s, t) | X Ao ) 
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as n — > oo. Clearly 

E(|E(S Ao (s,i) | W)-u n (s,t)\ |X Ao ) < ^Var(5 Ao (s,t) |X Ao ). (8.5) 

Recall that, 

S Ao ( S ,i):= £ (X 7 -X 7 (W A °)) 

where 

y x (s, t) = { 7 g ^ n (fc) 1 |a n 7 | = (k - s , k - 1)}. 

We start with the term T\. We consider different cases depending on the values of s,t. 
Note that, W,(S\ (s,t) \ X Ao ) = u n (s,t) <C o 2 j [i for st < k 2 . Thus, heuristically for st < k 2 , 
the contribution in Ti should be <C 1 as n — > 00. Obviously the nontrivial case is when 
s = t = k. 



Case 1. st > 0, s + t < 2k — 2: In this case we have u n (s, t) > and thus 

E(X 7 (W A ») |X Ao ) <E(X 7 |X Ao ) 
for 7 G J^ Ao (s,t). Now we have 

E(|E(Sao(M) I W)-u n (s,t)\ |X Ao ) 

< J] E(X 7 +X 7 (W Ao ) |X Ao ) + |n„(s,t)| 

= E(X 7 +X 7 (W A «)|X Ao )+ £ E(X 7 -X 7 (W A °)|X Ao ) 

76^A (s,t) 76^A (s,*) 

J (t) ^ s (" f J P ( X [«+l.«+fe]x[t+l,t+fc] I X Wx[fc])- 
Now using the results in case 3 and 5 from the proof of Theorem 3.9 we have 

m^+mw+k] I %x W ) < n- fe + 2fe ^-)( fe -*)/( 2fe2 -*)+°( 1 ) (8.6) 



and 



( k\ ( k\ ( n — k\ (n — k\ , 2 / 
\s/W/\ s A t ) ^ [ s + 1 ' s + fe ] x [*+ 1 .*+ fc ] I W x [ fe ^ - £nCJ 



where 

fc — 1 

e n := 0((log n/n 2 ) (fc+D(^+2fc-D (l og n) 2 ^ 1 ). 

Thus we have 

E(| E(S\ Q (s,t) I W) - u n (s,t)\ |X Ao )< £n . 

Case 2: s + £ = 2fc — 1: This corresponds to the set of matrices which have exactly one 
row in common with Ao and no columns, or vice-vera. Without loss of generality assume 
the former case (the later is dealt with identically) so that s = k — 1,£ = k. By (8.5) it is 
enough to prove that 

E(S Ao (A:-l,A;) 2 | X Ao ) < <r 4 //A 

Note that 

B(S\ (k -l,k)\ X Ao ) = v n (k - 1, k)/n < cr 2 //x. 
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We will write 

X 7 :=X 7 (W A «) and P Ao Q = P(- | X Ao ). 

Note that, any matrix in Sf\ {k — 1, k) is contained in the sub matrix [n] x [k + l,n] 
with exactly one row with index in [k]. For two matrix indices 7, 7' E ^\ {k — 1, k) define 
■A/"(7) 7') = (^j r ' c ) where I = 1 if 7, 7' share a row in [fc] and otherwise; r is the number of 
common rows between 7, 7' in [k + 1, n] and c is the number of common columns between 
7, 7'. Note that 

|{(7,7 / )|AA( 7 ,7 / ) = (Ar,c)}| 

=*«t-i W =o} + i {< =i}, (-*)(*)(--*; 

n - fc\ /fc - 1\ /n - 2k + 1 
fc — 1/ V r J \ k — 1 — r 



4fc— 2-r-c\ 



= 0(n 
Thus we have 

E(5Ao(*!-l,fc) 2 |X Ao ) 

< O k (l)j^j2n 4k - 2 - r - c n^-i,){^ ~Kr,) I 

£=0 r=0 c=0 

where Af(j, 7g,r, c ) = (^ r ) c )- Now for 7,7' G >y n (k) we have 
E((X 7 — Xy) ■ (Xy — Xy ) I X Ao ) 

= J^Ao (X 7 X^X 7 /Xy ) — PA (X 7 X T XyX 7 /) - P Ao (X r X 7 XyXy) + P Ao (X T X 7 XyXy). 

For rc = 0,r + c > 1 the contribution in ~E(S\ (k — l,k) 2 \ X Ao ) is bounded by 
Ofc(l)n 4fe_2_r_c n _2fc < Ofc(l)n 2fc_4 . To see this, consider the first term in the above 
equation. Here we require both 7, 7' to be locally optimal, in particular column optimal 
and thus must possess the largest k row sums in their respective column set, each of which 
has probability (even conditioning on X Ao ) of at most than 1/ ("l. 2 )■ When r + c = 0, one 
can prove that (using the method used in the proof of Theorem 3.9 for s = t = k) 

E((X 7 -X 7 )(X 7 ^ c -X^J I X Ao ) = 0(n- 2k - 2 ) 

and for r + c = 1 

E((X 7 -X 7 )(X 7 ^ c -X 7 ^J I X Ao ) = 0{n- 2k - 1 ). 

The n~ 2k term comes from the probability that both 7 and 7£ 5 r,c are locally optimal and 
the 1/n improvement is coming from the fact that E(X 7 — X 7 | X Ao ) = 0(n _fe_1 ). Thus 
for rc = 0, the total contribution is 0(n 2k ~ 4 ). When rc G [l,k(k — !)],£ G {0,1}, the 
contribution is 

„, . 2fe(r+^)c 

,4fc-2-r-c ~ /K +;j7T: 



n iK-*-r-c n — ' afc a -(fc-r-i)(fc-c) _ 

The maximum power occurs for r = c = f= lso that the contribution is bounded by 



„, , . 2k(l+e) 
n 2k -^+ 2k -2_ (k _ 1 _ e)(k _ 1) < n 2fc-4+4fc/(fc 2 +3A ; -2) 
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and 4k/(k 2 + 3k - 2) = 4/(Jfe + 1) - 8(k - l)/((jfe + 1)(> 2 + 3A; - 2)). Thus combining 
everything we have 

E(5 Ao (A; - 1, kf \1 Xq ) = 0( n 2fc-4 + 4/( fe+ l)-8( fc -l)/(( fe+ l)(^+3^2)) ) 

= 0{n- s{k - 1)/{{k+1) ( k2+?lk - 2)) ■ a A /n 2 ). 

Case 3. s = t = k. This corresponds to the set of matrices which have no common rows 
or columns with Ao- We move to the proof of 

Vax(S Xo (k,k)\l Xo ) «a 4 //A 

Note that, any matrix in y\ (k,k) is contained in the sub matrix [k + l,n] x [k + l,n]. 
Also we have 

|{7,7'e^(m I 1-ynVl = (r,c)}| 

"^)C)(7- 2 :)( n ; fc )0(r?)-(^)- 

Thus we have 

Var(S Ao (fc,fc) 2 \1\ ) 
l fc-i fe 

^ E E E n4fc_r_C CoV ^ - V7ne - ^>,c I ^Ao) 

^=0 r=0 c=0 

where M (7, 7 rjC ) = (r, c). For r = c = k, the total contribution in the variance is 

Ofr 2 **-*- 1 ) < 0(n 2fc - 4+3 /( fc+1 )). 

Note that here n - ^ 1 term comes from the fact that X 7 has probability n~ k and after 
changing the elements in the first k rows and k columns 7 is no longer locally optimal 
implies one of the new rows or columns beat 7 which has probability 1/n. In particular, 
similar to the variance calculation for L n , for all rc = 0, r + c > 1 the contribution is 

and for all rc > 1 the contribution is 

n 4fc-r-c n -2fc+2fcrc/(2fc 2 -(fc-r)(fc-c))-2/(l+max{r,c}/fc) 
< n -2(*-l)/((fc+l)(^+2k-l)) ff 4/ 2 

where the largest exponent occurs for r = c = 1. Thus the only terms remaining to bound 
are when r + c = 1 and r + c = 0. We look at the r + c = case first. We want to bound 

^ COV(Xy — Xy,Xy/ — Xy' | X\ Q ) . 

7,7'G.rA (fc,fc),| 7 n7'|=(o,o) 

Number of summands in the above sum is 0(n 4fc ). Now after some simplification it is easy 
to see that we need to bound 

Cov(X 7 X^,X 7 /X^ | X Ao ) 
which, by Lemma 3.10 can be bounded by 

n -2fc-2-2fe/(fc+l) = n -2fc-4+2/(fe+l) 
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Thus the total contribution is 

n 4fc-2fe-4+2/(fc+l) = n 2fe-4+4/(fc+l)-2/(fc+l) = n -2/(fc+l) (J 4 ^2 

Similarly for the r = 1, c = case the total contribution is 

n 4fc-l n -2fc-2-l = n 2fc-4 = n -4/(fc+l) (J 4 /M 2_ 

Combining everything we have T± — > as n — > oo. 
Now we show that 

k k 



r2 = EE\/A Var (' s Ao(^i)l^o) 



s=l t=l 

as n — > oo. Note that, E(S\ (s,i) | 2a ) = u n{s,t) < a 2 / ' ll for all s,t. Heuristically for 
fixed s, t the contribution in T2 should be < ^//i/a 3 ■ cr^/fi 2 = y/cr/n — > as n — > oo. We 
leave the proof to the interested reader where the proof follows exactly the same steps used 
in case 1 — 3 of the proof of T\ — > 0. Combining everything finally we have the result that 

d w (L,N(0,l))->0 (8.7) 
as n — > oo. ■ 
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